You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by marco <me...@gmail.com> on 2021/10/20 16:18:39 UTC
Flink on kubernetes HA ::Renew deadline reached
Hello flink community::
I am deploying flink application cluster standalone mode on kubernetes, but i am facing some problems
the job starts normally and it continues to run but at some point in time it crushes and gets restarted.
Does anyone facing the same problem or know how to resolve it.
I am using flink 1.12.5 version.
here's the log of the JM:
{"@timestamp":"2021-10-19T04:47:29.887+02:00","@version":"1","message":"WebSocket successfully opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:49:06.833+02:00","@version":"1","message":"Triggering checkpoint 2930 (type=CHECKPOINT) @ 1634611746678 for job 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"Checkpoint Timer","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:49:07.000+02:00","@version":"1","message":"Completed checkpoint 2930 for job 00000000000000000000000000000000 (474 bytes in 229 ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:51:40.861+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Submitting reconnect task to the executor","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Scheduling reconnect task","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor for Watch 1899640261","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Current reconnect backoff is 1000 milliseconds (T0)","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor for Watch 1899640261","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:41.863+02:00","@version":"1","message":"Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor for Watch 1899640261","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:41.864+02:00","@version":"1","message":"WebSocket successfully opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.916+02:00","@version":"1","message":"Renew deadline reached after 120 seconds while renewing lock ConfigMapLock: flink-dev - job1-dispatcher-leader (8f7dddbb-236a-43fe-a706-1795b494838d)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-19-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.918+02:00","@version":"1","message":"Stopping SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping dispatcher akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping all currently running jobs of dispatcher akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping the JobMaster for job flinkJob(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Stopping KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@367fad26","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@28d6d80e","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Job flinkJob (00000000000000000000000000000000) switched from state RUNNING to SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException: Stopping JobMaster for job flinkJob(00000000000000000000000000000000).
org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:413)
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
akka.actor.Actor.aroundReceive(Actor.scala:517)
akka.actor.Actor.aroundReceive$(Actor.scala:515)
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
akka.actor.ActorCell.invoke(ActorCell.scala:561)
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
akka.dispatch.Mailbox.run(Mailbox.scala:225)
akka.dispatch.Mailbox.exec(Mailbox.scala:235)
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"}
{"@timestamp":"2021-10-19T04:53:16.925+02:00","@version":"1","message":"Source: Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.926+02:00","@version":"1","message":"Source: Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.927+02:00","@version":"1","message":"Discarding the results produced by task execution 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.928+02:00","@version":"1","message":"Discarding the results produced by task execution 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding the results produced by task execution e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding the results produced by task execution e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding the results produced by task execution 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding the results produced by task execution 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Filter with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding the results produced by task execution 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding the results produced by task execution 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding the results produced by task execution 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding the results produced by task execution 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding the results produced by task execution 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding the results produced by task execution 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding the results produced by task execution f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding the results produced by task execution f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding the results produced by task execution a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding the results produced by task execution a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding the results produced by task execution 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding the results produced by task execution 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"httpAd call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding the results produced by task execution 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding the results produced by task execution 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding the results produced by task execution 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding the results produced by task execution 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding the results produced by task execution 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding the results produced by task execution 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Update Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Discarding the results produced by task execution 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.982+02:00","@version":"1","message":"Discarding the results produced by task execution 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink: Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink: Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.984+02:00","@version":"1","message":"Discarding the results produced by task execution 4467a95a23f6a5e983f006fb12832b10.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink: Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink: Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Discarding the results produced by task execution cf5ce6a12a584da4651a7d341cb1aeef.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Stopping checkpoint coordinator for job 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Shutting down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Job 00000000000000000000000000000000 has been suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Suspending SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Close ResourceManager connection 81a0f3b34590e11df305b58878a355b1: Stopping JobMaster for job flinkJob(00000000000000000000000000000000)..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Stopping SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Disconnect job manager bc901e3d25d406f70027b81673c742ba@akka.tcp://flink@172.24.10.160:6123/user/rpc/jobmanager_2 for job 00000000000000000000000000000000 from the resource manager.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Leader election started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Stopping DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Attempting to acquire leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@4d181edd","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5efd28bd","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Loop thread interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException: null
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Job 00000000000000000000000000000000 reached terminal state SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.996+02:00","@version":"1","message":"Released job graph 00000000000000000000000000000000 from KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-4","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.997+02:00","@version":"1","message":"Application failed unexpectedly: ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-4","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException: null
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:16.998+02:00","@version":"1","message":"Fatal error occurred in the cluster entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-4","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException: Application failed unexpectedly.
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CancellationException: null
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
\t... 22 common frames omitted
"}
{"@timestamp":"2021-10-19T04:53:17.084+02:00","@version":"1","message":"Shutting StandaloneApplicationClusterEntryPoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint shutdown hook","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.085+02:00","@version":"1","message":"Shutting down rest endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint shutdown hook","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.086+02:00","@version":"1","message":"Stopped BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer shutdown hook","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Successfully Acquired leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader (8f7dddbb-236a-43fe-a706-1795b494838d)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-25081-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Attempting to renew leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.185+02:00","@version":"1","message":"Removing cache directory /tmp/flink-web-682b3e4f-ed72-4b00-96a0-da5c8248b62c/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Stopping DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6f9e08d4","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@42c1c89c","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Loop thread interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException: null
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Shut down complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@11787b64","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@4c063eb4","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@4839a3ec","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@202d9236","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@d3b348f","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Loop thread interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException: null
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Stopping SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Closing the SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Suspending the SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Stopping DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7be008e9","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Loop thread interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException: null
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@57ae1976","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3f6e134d","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@2a1892aa","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
Re: Flink on kubernetes HA ::Renew deadline reached
Posted by Xintong Song <to...@gmail.com>.
Sorry for being ambiguous. I didn't mean to suggest you use the native K8s
HA. I meant I noticed you're using the native K8s HA from your logs.
Currently, Flink supports two HA implementations: Zookeeper-based and
Kubernetes ConfigMap -based. The Kubernetes ConfigMap -based HA is also
called native K8s HA, which is available to both standalone and native
Kubernetes deployments.
Thank you~
Xintong Song
On Mon, Oct 25, 2021 at 6:33 PM marco <me...@gmail.com> wrote:
> Hello thanks for your response,
>
> I just want to know why you suggest that i should use the native
> Kubernetes HA.
> Im my case I need to use the standalone application mode.
>
> On 2021/10/25 09:25:21, Xintong Song <to...@gmail.com> wrote:
> > You should be using the native Kubernetes HA. This error message suggests
> > Flink's dispatcher cannot successfully renew it's leadership within 120s,
> > so according to the leader election protocol it loses the leadership and
> > thus shutted down.
> >
> > This is usually caused by:
> > 1) Long time GC pause in Flink's master process. You can check the
> > jobmanager GC logs to verify that. And if it is indeed the problem, you
> can
> > try to increase jobmanager's heap memory size.
> > 2) Kubernetes API server is temporarily not accessible. You would need to
> > check the Kubernetes logs, and if it is indeed the problem, you can
> either
> > try to fix the API server accessibility or increase the leadership renew
> > timeout (high-availability.kubernetes.leader-election.renew-deadline).
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Oct 25, 2021 at 5:08 PM marco <me...@gmail.com> wrote:
> >
> > > Any suggestions would be appreciated.
> > >
> > > On 2021/10/20 16:18:39, marco <me...@gmail.com> wrote:
> > > >
> > > >
> > > > Hello flink community::
> > > >
> > > > I am deploying flink application cluster standalone mode on
> kubernetes,
> > > but i am facing some problems
> > > >
> > > > the job starts normally and it continues to run but at some point in
> > > time it crushes and gets restarted.
> > > >
> > > > Does anyone facing the same problem or know how to resolve it.
> > > >
> > > > I am using flink 1.12.5 version.
> > > >
> > > > here's the log of the JM:
> > > >
> > > >
> > >
> {"@timestamp":"2021-10-19T04:47:29.887+02:00","@version":"1","message":"WebSocket
> > > successfully
> > >
> opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:49:06.833+02:00","@version":"1","message":"Triggering
> > > checkpoint 2930 (type=CHECKPOINT) @ 1634611746678 for job
> > >
> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"Checkpoint
> > > Timer","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:49:07.000+02:00","@version":"1","message":"Completed
> > > checkpoint 2930 for job 00000000000000000000000000000000 (474 bytes in
> 229
> > >
> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:51:40.861+02:00","@version":"1","message":"WebSocket
> > > close received. code: 1000, reason:
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > WebSocket https://172.31.64.1/..
> .","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Submitting
> > > reconnect task to the
> > >
> executor","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > WebSocket https://172.31.64.1/..
> .","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Scheduling
> > > reconnect
> > >
> task","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
> > > for Watch 1899640261","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Current
> > > reconnect backoff is 1000 milliseconds
> > >
> (T0)","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
> > > for Watch 1899640261","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:51:41.863+02:00","@version":"1","message":"Connecting
> > > websocket ...
> > >
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor
> > > for Watch 1899640261","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:51:41.864+02:00","@version":"1","message":"WebSocket
> > > successfully
> > >
> opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.916+02:00","@version":"1","message":"Renew
> > > deadline reached after 120 seconds while renewing lock ConfigMapLock:
> > > flink-dev - job1-dispatcher-leader
> > >
> (8f7dddbb-236a-43fe-a706-1795b494838d)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-19-thread-1","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.918+02:00","@version":"1","message":"Stopping
> > >
> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
> > > dispatcher akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1
> > >
> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
> > > all currently running jobs of dispatcher akka.tcp://
> > > flink@172.24.10.160:6123/user/rpc/dispatcher_1
> > >
> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
> > > the JobMaster for job
> > >
> flinkJob(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
> > >
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Stopping
> > >
> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Force
> > > closing the watch
> > >
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@367fad26
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"The
> > > watcher is
> > >
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Closing
> > > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > > .RealWebSocket@28d6d80e
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"WebSocket
> > > close received. code: 1000, reason:
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Ignoring
> > > onClose for already closed/closing
> > >
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Job
> > > flinkJob (00000000000000000000000000000000) switched from state
> RUNNING to
> > >
> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
> > > Stopping JobMaster for job flinkJob(00000000000000000000000000000000).
> > > >
> org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:413)
> > > >
> > >
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> > > >
> > >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
> > > >
> > >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
> > > > akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> > > > akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> > > > scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> > > > scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> > > > akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> > > > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> > > > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> > > > akka.actor.Actor.aroundReceive(Actor.scala:517)
> > > > akka.actor.Actor.aroundReceive$(Actor.scala:515)
> > > > akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> > > > akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> > > > akka.actor.ActorCell.invoke(ActorCell.scala:561)
> > > > akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> > > > akka.dispatch.Mailbox.run(Mailbox.scala:225)
> > > > akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> > > > akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > >
> > >
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> > > >
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > >
> > >
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > "}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.925+02:00","@version":"1","message":"Source:
> > > Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from
> RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.926+02:00","@version":"1","message":"Source:
> > > Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from
> CANCELING
> > > to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.927+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.928+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
> > > conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from
> RUNNING
> > > to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
> > > conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from
> CANCELING
> > > to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
> > > conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from
> RUNNING
> > > to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
> > > conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from
> CANCELING
> > > to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Filter
> > > with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from
> > > RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> > > with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> > > with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from
> > > RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> > > with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
> > > Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from
> > > RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
> > > Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
> > > Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from
> > > RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
> > > Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
> > > Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from
> > > RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
> > > Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
> > > Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from
> > > RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
> > > Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"httpAd
> > > call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> > > call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from CANCELING
> to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> > > call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> > > call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from CANCELING
> to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> > > Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from
> RUNNING
> > > to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> > > Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> > > Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from
> RUNNING
> > > to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Update
> > > Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.982+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
> > > Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from
> > > RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
> > > Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.984+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> 4467a95a23f6a5e983f006fb12832b10.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
> > > Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from
> > > RUNNING to
> > >
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
> > > Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from
> > > CANCELING to
> > >
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Discarding
> > > the results produced by task execution
> > >
> cf5ce6a12a584da4651a7d341cb1aeef.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Stopping
> > > checkpoint coordinator for job
> > >
> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Shutting
> > >
> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Job
> > > 00000000000000000000000000000000 has been
> > >
> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Suspending
> > >
> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Close
> > > ResourceManager connection 81a0f3b34590e11df305b58878a355b1: Stopping
> > > JobMaster for job
> > >
> flinkJob(00000000000000000000000000000000)..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Stopping
> > >
> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Disconnect
> > > job manager bc901e3d25d406f70027b81673c742ba@akka.tcp://
> > > flink@172.24.10.160:6123/user/rpc/jobmanager_2 for job
> > > 00000000000000000000000000000000 from the resource
> > >
> manager.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Leader
> > > election
> > >
> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Stopping
> > >
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Attempting
> > > to acquire leader lease 'ConfigMapLock: flink-dev -
> job1-dispatcher-leader
> > >
> (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
> > >
> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Force
> > > closing the watch
> > >
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@4d181edd
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"The
> > > watcher is
> > >
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
> > > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > > .RealWebSocket@5efd28bd
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Loop
> > > thread
> > >
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> > > null
> > > >
> > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> > > >
> > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > > > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > java.lang.Thread.run(Thread.java:748)
> > > > "}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"WebSocket
> > > close received. code: 1000, reason:
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Job
> > > 00000000000000000000000000000000 reached terminal state
> > >
> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Ignoring
> > > onClose for already closed/closing
> > >
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.996+02:00","@version":"1","message":"Released
> > > job graph 00000000000000000000000000000000 from
> > >
> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-4","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.997+02:00","@version":"1","message":"Application
> > > failed unexpectedly:
> > >
> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-4","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
> > > null
> > > >
> > >
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
> > > >
> > >
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
> > > >
> > >
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
> > > >
> > >
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> > > >
> > >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > > >
> > >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> > > >
> > >
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> > > >
> > >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > > >
> > >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > > >
> > >
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> > > >
> > >
> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
> > > >
> > >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > > >
> > >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > > >
> > >
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
> > > >
> > >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > java.lang.Thread.run(Thread.java:748)
> > > > "}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:16.998+02:00","@version":"1","message":"Fatal
> > > error occurred in the cluster
> > >
> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-4","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
> > > Application failed unexpectedly.
> > > >
> > >
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
> > > >
> > >
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> > > >
> > >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > > >
> > >
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
> > > >
> > >
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
> > > >
> > >
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
> > > >
> > >
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> > > >
> > >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > > >
> > >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> > > >
> > >
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> > > >
> > >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > > >
> > >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > > >
> > >
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> > > >
> > >
> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
> > > >
> > >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > > >
> > >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > > >
> > >
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> > > >
> > >
> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
> > > >
> > >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > java.lang.Thread.run(Thread.java:748)
> > > > Caused by: java.util.concurrent.CancellationException: null
> > > >
> > >
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
> > > > \t... 22 common frames omitted
> > > > "}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.084+02:00","@version":"1","message":"Shutting
> > > StandaloneApplicationClusterEntryPoint down with application status
> > > UNKNOWN. Diagnostics Cluster entrypoint has been closed
> > >
> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
> > > shutdown hook","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.085+02:00","@version":"1","message":"Shutting
> > > down rest
> > >
> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
> > > shutdown hook","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.086+02:00","@version":"1","message":"Stopped
> > > BLOB server at 0.0.0.0:6124
> ","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
> > > shutdown hook","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Successfully
> > > Acquired leader lease 'ConfigMapLock: flink-dev -
> job1-dispatcher-leader
> > >
> (8f7dddbb-236a-43fe-a706-1795b494838d)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-25081-thread-1","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Attempting
> > > to renew leader lease 'ConfigMapLock: flink-dev -
> job1-dispatcher-leader
> > >
> (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.185+02:00","@version":"1","message":"Removing
> > > cache directory
> > >
> /tmp/flink-web-682b3e4f-ed72-4b00-96a0-da5c8248b62c/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Stopping
> > >
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> > >
> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Force
> > > closing the watch
> > >
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6f9e08d4
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"The
> > > watcher is
> > >
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> > > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > > .RealWebSocket@42c1c89c
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Loop
> > > thread
> > >
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> > > null
> > > >
> > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> > > >
> > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > > > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > java.lang.Thread.run(Thread.java:748)
> > > > "}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Shut
> > > down
> > >
> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> > >
> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
> > >
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
> > >
> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"WebSocket
> > > close received. code: 1000, reason:
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Force
> > > closing the watch
> > >
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@11787b64
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Ignoring
> > > onClose for already closed/closing
> > >
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"The
> > > watcher is
> > >
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Closing
> > > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > > .RealWebSocket@4c063eb4
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
> > >
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
> > >
> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
> > > closing the watch
> > >
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"The
> > > watcher is
> > >
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
> > > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > > .RealWebSocket@4839a3ec
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
> > > close received. code: 1000, reason:
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Ignoring
> > > onClose for already closed/closing
> > >
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
> > >
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
> > >
> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
> > > closing the watch
> > >
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@202d9236
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"The
> > > watcher is
> > >
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
> > > close received. code: 1000, reason:
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Closing
> > > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > > .RealWebSocket@d3b348f
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Ignoring
> > > onClose for already closed/closing
> > >
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Loop
> > > thread
> > >
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> > > null
> > > >
> > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> > > >
> > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > > > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > java.lang.Thread.run(Thread.java:748)
> > > > "}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Stopping
> > >
> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"WebSocket
> > > close received. code: 1000, reason:
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Ignoring
> > > onClose for already closed/closing
> > >
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Closing
> > > the
> > >
> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Suspending
> > > the
> > >
> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Stopping
> > >
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
> > >
> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Force
> > > closing the watch
> > >
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7be008e9
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"The
> > > watcher is
> > >
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Loop
> > > thread
> > >
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> > > null
> > > >
> > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> > > >
> > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > > > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> > > >
> > >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > java.lang.Thread.run(Thread.java:748)
> > > > "}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
> > > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > > .RealWebSocket@57ae1976
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
> > >
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
> > >
> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Force
> > > closing the watch
> > >
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3f6e134d
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"WebSocket
> > > close received. code: 1000, reason:
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"The
> > > watcher is
> > >
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
> > > onClose for already closed/closing
> > >
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Closing
> > > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > > .RealWebSocket@2a1892aa
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"WebSocket
> > > close received. code: 1000, reason:
> > >
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > >
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
> > > onClose for already closed/closing
> > >
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > > >
> > > >
> > >
> >
>
Re: Flink on kubernetes HA ::Renew deadline reached
Posted by marco <me...@gmail.com>.
Hello thanks for your response,
I just want to know why you suggest that i should use the native Kubernetes HA.
Im my case I need to use the standalone application mode.
On 2021/10/25 09:25:21, Xintong Song <to...@gmail.com> wrote:
> You should be using the native Kubernetes HA. This error message suggests
> Flink's dispatcher cannot successfully renew it's leadership within 120s,
> so according to the leader election protocol it loses the leadership and
> thus shutted down.
>
> This is usually caused by:
> 1) Long time GC pause in Flink's master process. You can check the
> jobmanager GC logs to verify that. And if it is indeed the problem, you can
> try to increase jobmanager's heap memory size.
> 2) Kubernetes API server is temporarily not accessible. You would need to
> check the Kubernetes logs, and if it is indeed the problem, you can either
> try to fix the API server accessibility or increase the leadership renew
> timeout (high-availability.kubernetes.leader-election.renew-deadline).
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Oct 25, 2021 at 5:08 PM marco <me...@gmail.com> wrote:
>
> > Any suggestions would be appreciated.
> >
> > On 2021/10/20 16:18:39, marco <me...@gmail.com> wrote:
> > >
> > >
> > > Hello flink community::
> > >
> > > I am deploying flink application cluster standalone mode on kubernetes,
> > but i am facing some problems
> > >
> > > the job starts normally and it continues to run but at some point in
> > time it crushes and gets restarted.
> > >
> > > Does anyone facing the same problem or know how to resolve it.
> > >
> > > I am using flink 1.12.5 version.
> > >
> > > here's the log of the JM:
> > >
> > >
> > {"@timestamp":"2021-10-19T04:47:29.887+02:00","@version":"1","message":"WebSocket
> > successfully
> > opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:49:06.833+02:00","@version":"1","message":"Triggering
> > checkpoint 2930 (type=CHECKPOINT) @ 1634611746678 for job
> > 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"Checkpoint
> > Timer","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:49:07.000+02:00","@version":"1","message":"Completed
> > checkpoint 2930 for job 00000000000000000000000000000000 (474 bytes in 229
> > ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:51:40.861+02:00","@version":"1","message":"WebSocket
> > close received. code: 1000, reason:
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Submitting
> > reconnect task to the
> > executor","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Scheduling
> > reconnect
> > task","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
> > for Watch 1899640261","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Current
> > reconnect backoff is 1000 milliseconds
> > (T0)","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
> > for Watch 1899640261","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:51:41.863+02:00","@version":"1","message":"Connecting
> > websocket ...
> > io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor
> > for Watch 1899640261","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:51:41.864+02:00","@version":"1","message":"WebSocket
> > successfully
> > opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.916+02:00","@version":"1","message":"Renew
> > deadline reached after 120 seconds while renewing lock ConfigMapLock:
> > flink-dev - job1-dispatcher-leader
> > (8f7dddbb-236a-43fe-a706-1795b494838d)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-19-thread-1","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.918+02:00","@version":"1","message":"Stopping
> > SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
> > dispatcher akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1
> > .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
> > all currently running jobs of dispatcher akka.tcp://
> > flink@172.24.10.160:6123/user/rpc/dispatcher_1
> > .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
> > the JobMaster for job
> > flinkJob(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
> > DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Stopping
> > KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Force
> > closing the watch
> > io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@367fad26
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"The
> > watcher is
> > closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Closing
> > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > .RealWebSocket@28d6d80e
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"WebSocket
> > close received. code: 1000, reason:
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Ignoring
> > onClose for already closed/closing
> > websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Job
> > flinkJob (00000000000000000000000000000000) switched from state RUNNING to
> > SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
> > Stopping JobMaster for job flinkJob(00000000000000000000000000000000).
> > > org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:413)
> > >
> > org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> > >
> > org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
> > >
> > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
> > > akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> > > akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> > > scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> > > scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> > > akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> > > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> > > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> > > akka.actor.Actor.aroundReceive(Actor.scala:517)
> > > akka.actor.Actor.aroundReceive$(Actor.scala:515)
> > > akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> > > akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> > > akka.actor.ActorCell.invoke(ActorCell.scala:561)
> > > akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> > > akka.dispatch.Mailbox.run(Mailbox.scala:225)
> > > akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> > > akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > >
> > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> > > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > >
> > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > "}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.925+02:00","@version":"1","message":"Source:
> > Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.926+02:00","@version":"1","message":"Source:
> > Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from CANCELING
> > to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.927+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.928+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
> > conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from RUNNING
> > to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
> > conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from CANCELING
> > to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
> > conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from RUNNING
> > to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
> > conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from CANCELING
> > to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Filter
> > with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from
> > RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> > with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> > with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from
> > RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> > with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
> > Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from
> > RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
> > Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
> > Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from
> > RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
> > Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
> > Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from
> > RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
> > Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
> > Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from
> > RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
> > Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"httpAd
> > call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> > call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> > call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> > call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> > Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from RUNNING
> > to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> > Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> > Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from RUNNING
> > to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Update
> > Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.982+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
> > Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from
> > RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
> > Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.984+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > 4467a95a23f6a5e983f006fb12832b10.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
> > Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from
> > RUNNING to
> > CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
> > Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from
> > CANCELING to
> > CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Discarding
> > the results produced by task execution
> > cf5ce6a12a584da4651a7d341cb1aeef.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Stopping
> > checkpoint coordinator for job
> > 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Shutting
> > down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Job
> > 00000000000000000000000000000000 has been
> > suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Suspending
> > SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Close
> > ResourceManager connection 81a0f3b34590e11df305b58878a355b1: Stopping
> > JobMaster for job
> > flinkJob(00000000000000000000000000000000)..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Stopping
> > SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Disconnect
> > job manager bc901e3d25d406f70027b81673c742ba@akka.tcp://
> > flink@172.24.10.160:6123/user/rpc/jobmanager_2 for job
> > 00000000000000000000000000000000 from the resource
> > manager.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Leader
> > election
> > started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Stopping
> > DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Attempting
> > to acquire leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
> > (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
> > KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Force
> > closing the watch
> > io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@4d181edd
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"The
> > watcher is
> > closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
> > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > .RealWebSocket@5efd28bd
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Loop
> > thread
> > interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> > null
> > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > java.lang.Thread.run(Thread.java:748)
> > > "}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"WebSocket
> > close received. code: 1000, reason:
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Job
> > 00000000000000000000000000000000 reached terminal state
> > SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Ignoring
> > onClose for already closed/closing
> > websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.996+02:00","@version":"1","message":"Released
> > job graph 00000000000000000000000000000000 from
> > KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-4","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.997+02:00","@version":"1","message":"Application
> > failed unexpectedly:
> > ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-4","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
> > null
> > >
> > java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
> > >
> > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
> > >
> > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
> > >
> > org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> > >
> > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> > >
> > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > >
> > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> > >
> > org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> > >
> > java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> > >
> > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> > >
> > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > >
> > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > >
> > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> > >
> > org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
> > >
> > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> > >
> > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > >
> > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > >
> > java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> > >
> > java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
> > >
> > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > java.lang.Thread.run(Thread.java:748)
> > > "}
> > >
> > {"@timestamp":"2021-10-19T04:53:16.998+02:00","@version":"1","message":"Fatal
> > error occurred in the cluster
> > entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-4","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
> > Application failed unexpectedly.
> > >
> > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
> > >
> > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> > >
> > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> > >
> > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > >
> > java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
> > >
> > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
> > >
> > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
> > >
> > org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> > >
> > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> > >
> > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > >
> > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> > >
> > org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> > >
> > java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> > >
> > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> > >
> > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > >
> > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > >
> > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> > >
> > org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
> > >
> > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> > >
> > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> > >
> > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> > >
> > java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> > >
> > java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
> > >
> > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > java.lang.Thread.run(Thread.java:748)
> > > Caused by: java.util.concurrent.CancellationException: null
> > >
> > java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
> > > \t... 22 common frames omitted
> > > "}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.084+02:00","@version":"1","message":"Shutting
> > StandaloneApplicationClusterEntryPoint down with application status
> > UNKNOWN. Diagnostics Cluster entrypoint has been closed
> > externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
> > shutdown hook","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.085+02:00","@version":"1","message":"Shutting
> > down rest
> > endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
> > shutdown hook","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.086+02:00","@version":"1","message":"Stopped
> > BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
> > shutdown hook","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Successfully
> > Acquired leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
> > (8f7dddbb-236a-43fe-a706-1795b494838d)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-25081-thread-1","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Attempting
> > to renew leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
> > (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.185+02:00","@version":"1","message":"Removing
> > cache directory
> > /tmp/flink-web-682b3e4f-ed72-4b00-96a0-da5c8248b62c/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Stopping
> > DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> > KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Force
> > closing the watch
> > io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6f9e08d4
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"The
> > watcher is
> > closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > .RealWebSocket@42c1c89c
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Loop
> > thread
> > interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> > null
> > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > java.lang.Thread.run(Thread.java:748)
> > > "}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Shut
> > down
> > complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> > components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
> > DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
> > KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"WebSocket
> > close received. code: 1000, reason:
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Force
> > closing the watch
> > io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@11787b64
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Ignoring
> > onClose for already closed/closing
> > websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"The
> > watcher is
> > closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Closing
> > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > .RealWebSocket@4c063eb4
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
> > DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
> > KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
> > closing the watch
> > io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"The
> > watcher is
> > closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
> > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > .RealWebSocket@4839a3ec
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
> > close received. code: 1000, reason:
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Ignoring
> > onClose for already closed/closing
> > websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
> > DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
> > KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
> > closing the watch
> > io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@202d9236
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"The
> > watcher is
> > closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
> > close received. code: 1000, reason:
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Closing
> > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > .RealWebSocket@d3b348f
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Ignoring
> > onClose for already closed/closing
> > websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Loop
> > thread
> > interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> > null
> > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > java.lang.Thread.run(Thread.java:748)
> > > "}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Stopping
> > SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-29","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"WebSocket
> > close received. code: 1000, reason:
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Ignoring
> > onClose for already closed/closing
> > websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Closing
> > the
> > SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Suspending
> > the
> > SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Stopping
> > DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
> > KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Force
> > closing the watch
> > io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7be008e9
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"The
> > watcher is
> > closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Loop
> > thread
> > interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> > null
> > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> > >
> > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > java.lang.Thread.run(Thread.java:748)
> > > "}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
> > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > .RealWebSocket@57ae1976
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
> > DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
> > KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Force
> > closing the watch
> > io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3f6e134d
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"WebSocket
> > close received. code: 1000, reason:
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"The
> > watcher is
> > closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
> > onClose for already closed/closing
> > websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Closing
> > websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> > .RealWebSocket@2a1892aa
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"WebSocket
> > close received. code: 1000, reason:
> > ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
> > onClose for already closed/closing
> > websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> > https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> > >
> > >
> >
>
Re: Flink on kubernetes HA ::Renew deadline reached
Posted by Xintong Song <to...@gmail.com>.
You should be using the native Kubernetes HA. This error message suggests
Flink's dispatcher cannot successfully renew it's leadership within 120s,
so according to the leader election protocol it loses the leadership and
thus shutted down.
This is usually caused by:
1) Long time GC pause in Flink's master process. You can check the
jobmanager GC logs to verify that. And if it is indeed the problem, you can
try to increase jobmanager's heap memory size.
2) Kubernetes API server is temporarily not accessible. You would need to
check the Kubernetes logs, and if it is indeed the problem, you can either
try to fix the API server accessibility or increase the leadership renew
timeout (high-availability.kubernetes.leader-election.renew-deadline).
Thank you~
Xintong Song
On Mon, Oct 25, 2021 at 5:08 PM marco <me...@gmail.com> wrote:
> Any suggestions would be appreciated.
>
> On 2021/10/20 16:18:39, marco <me...@gmail.com> wrote:
> >
> >
> > Hello flink community::
> >
> > I am deploying flink application cluster standalone mode on kubernetes,
> but i am facing some problems
> >
> > the job starts normally and it continues to run but at some point in
> time it crushes and gets restarted.
> >
> > Does anyone facing the same problem or know how to resolve it.
> >
> > I am using flink 1.12.5 version.
> >
> > here's the log of the JM:
> >
> >
> {"@timestamp":"2021-10-19T04:47:29.887+02:00","@version":"1","message":"WebSocket
> successfully
> opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:49:06.833+02:00","@version":"1","message":"Triggering
> checkpoint 2930 (type=CHECKPOINT) @ 1634611746678 for job
> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"Checkpoint
> Timer","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:49:07.000+02:00","@version":"1","message":"Completed
> checkpoint 2930 for job 00000000000000000000000000000000 (474 bytes in 229
> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:51:40.861+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Submitting
> reconnect task to the
> executor","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Scheduling
> reconnect
> task","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
> for Watch 1899640261","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Current
> reconnect backoff is 1000 milliseconds
> (T0)","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
> for Watch 1899640261","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:51:41.863+02:00","@version":"1","message":"Connecting
> websocket ...
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor
> for Watch 1899640261","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:51:41.864+02:00","@version":"1","message":"WebSocket
> successfully
> opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.916+02:00","@version":"1","message":"Renew
> deadline reached after 120 seconds while renewing lock ConfigMapLock:
> flink-dev - job1-dispatcher-leader
> (8f7dddbb-236a-43fe-a706-1795b494838d)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-19-thread-1","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.918+02:00","@version":"1","message":"Stopping
> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
> dispatcher akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1
> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
> all currently running jobs of dispatcher akka.tcp://
> flink@172.24.10.160:6123/user/rpc/dispatcher_1
> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
> the JobMaster for job
> flinkJob(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Stopping
> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@367fad26
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Closing
> websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> .RealWebSocket@28d6d80e
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Job
> flinkJob (00000000000000000000000000000000) switched from state RUNNING to
> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
> Stopping JobMaster for job flinkJob(00000000000000000000000000000000).
> > org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:413)
> >
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
> >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
> > akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> > akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> > scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> > scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> > akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> > akka.actor.Actor.aroundReceive(Actor.scala:517)
> > akka.actor.Actor.aroundReceive$(Actor.scala:515)
> > akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> > akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> > akka.actor.ActorCell.invoke(ActorCell.scala:561)
> > akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> > akka.dispatch.Mailbox.run(Mailbox.scala:225)
> > akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> > akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> >
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > "}
> >
> {"@timestamp":"2021-10-19T04:53:16.925+02:00","@version":"1","message":"Source:
> Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.926+02:00","@version":"1","message":"Source:
> Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from CANCELING
> to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.927+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.928+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
> conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from RUNNING
> to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
> conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from CANCELING
> to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
> conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from RUNNING
> to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
> conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from CANCELING
> to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Filter
> with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
> Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
> Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
> Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
> Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
> Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
> Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
> Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
> Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"httpAd
> call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from RUNNING
> to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from RUNNING
> to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Update
> Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.982+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
> Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
> Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.984+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 4467a95a23f6a5e983f006fb12832b10.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
> Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
> Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> cf5ce6a12a584da4651a7d341cb1aeef.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Stopping
> checkpoint coordinator for job
> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Shutting
> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Job
> 00000000000000000000000000000000 has been
> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Suspending
> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Close
> ResourceManager connection 81a0f3b34590e11df305b58878a355b1: Stopping
> JobMaster for job
> flinkJob(00000000000000000000000000000000)..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Stopping
> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Disconnect
> job manager bc901e3d25d406f70027b81673c742ba@akka.tcp://
> flink@172.24.10.160:6123/user/rpc/jobmanager_2 for job
> 00000000000000000000000000000000 from the resource
> manager.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Leader
> election
> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Stopping
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Attempting
> to acquire leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
> (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@4d181edd
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
> websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> .RealWebSocket@5efd28bd
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Loop
> thread
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> null
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > java.lang.Thread.run(Thread.java:748)
> > "}
> >
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Job
> 00000000000000000000000000000000 reached terminal state
> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:16.996+02:00","@version":"1","message":"Released
> job graph 00000000000000000000000000000000 from
> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-4","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:16.997+02:00","@version":"1","message":"Application
> failed unexpectedly:
> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-4","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
> null
> >
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
> >
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
> >
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
> >
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> >
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> >
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> >
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> >
> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
> >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> >
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> >
> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
> >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > java.lang.Thread.run(Thread.java:748)
> > "}
> >
> {"@timestamp":"2021-10-19T04:53:16.998+02:00","@version":"1","message":"Fatal
> error occurred in the cluster
> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-4","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
> Application failed unexpectedly.
> >
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
> >
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> >
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> >
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
> >
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
> >
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
> >
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> >
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> >
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> >
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> >
> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
> >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> >
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> >
> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
> >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > java.lang.Thread.run(Thread.java:748)
> > Caused by: java.util.concurrent.CancellationException: null
> >
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
> > \t... 22 common frames omitted
> > "}
> >
> {"@timestamp":"2021-10-19T04:53:17.084+02:00","@version":"1","message":"Shutting
> StandaloneApplicationClusterEntryPoint down with application status
> UNKNOWN. Diagnostics Cluster entrypoint has been closed
> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
> shutdown hook","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.085+02:00","@version":"1","message":"Shutting
> down rest
> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
> shutdown hook","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.086+02:00","@version":"1","message":"Stopped
> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
> shutdown hook","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Successfully
> Acquired leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
> (8f7dddbb-236a-43fe-a706-1795b494838d)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-25081-thread-1","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Attempting
> to renew leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
> (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.185+02:00","@version":"1","message":"Removing
> cache directory
> /tmp/flink-web-682b3e4f-ed72-4b00-96a0-da5c8248b62c/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Stopping
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6f9e08d4
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> .RealWebSocket@42c1c89c
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Loop
> thread
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> null
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > java.lang.Thread.run(Thread.java:748)
> > "}
> >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Shut
> down
> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@11787b64
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Closing
> websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> .RealWebSocket@4c063eb4
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
> websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> .RealWebSocket@4839a3ec
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@202d9236
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Closing
> websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> .RealWebSocket@d3b348f
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Loop
> thread
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> null
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > java.lang.Thread.run(Thread.java:748)
> > "}
> >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Stopping
> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-29","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Closing
> the
> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Suspending
> the
> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Stopping
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7be008e9
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Loop
> thread
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> null
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> >
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > java.lang.Thread.run(Thread.java:748)
> > "}
> >
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
> websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> .RealWebSocket@57ae1976
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3f6e134d
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> >
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Closing
> websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws
> .RealWebSocket@2a1892aa
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> >
> >
>
Re: Flink on kubernetes HA ::Renew deadline reached
Posted by marco <me...@gmail.com>.
Any suggestions would be appreciated.
On 2021/10/20 16:18:39, marco <me...@gmail.com> wrote:
>
>
> Hello flink community::
>
> I am deploying flink application cluster standalone mode on kubernetes, but i am facing some problems
>
> the job starts normally and it continues to run but at some point in time it crushes and gets restarted.
>
> Does anyone facing the same problem or know how to resolve it.
>
> I am using flink 1.12.5 version.
>
> here's the log of the JM:
>
> {"@timestamp":"2021-10-19T04:47:29.887+02:00","@version":"1","message":"WebSocket successfully opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:49:06.833+02:00","@version":"1","message":"Triggering checkpoint 2930 (type=CHECKPOINT) @ 1634611746678 for job 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"Checkpoint Timer","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:49:07.000+02:00","@version":"1","message":"Completed checkpoint 2930 for job 00000000000000000000000000000000 (474 bytes in 229 ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:51:40.861+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Submitting reconnect task to the executor","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Scheduling reconnect task","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor for Watch 1899640261","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Current reconnect backoff is 1000 milliseconds (T0)","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor for Watch 1899640261","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:41.863+02:00","@version":"1","message":"Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor for Watch 1899640261","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:41.864+02:00","@version":"1","message":"WebSocket successfully opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.916+02:00","@version":"1","message":"Renew deadline reached after 120 seconds while renewing lock ConfigMapLock: flink-dev - job1-dispatcher-leader (8f7dddbb-236a-43fe-a706-1795b494838d)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-19-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.918+02:00","@version":"1","message":"Stopping SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping dispatcher akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping all currently running jobs of dispatcher akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping the JobMaster for job flinkJob(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Stopping KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@367fad26","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@28d6d80e","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Job flinkJob (00000000000000000000000000000000) switched from state RUNNING to SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException: Stopping JobMaster for job flinkJob(00000000000000000000000000000000).
> org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:413)
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> akka.actor.Actor.aroundReceive(Actor.scala:517)
> akka.actor.Actor.aroundReceive$(Actor.scala:515)
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> akka.actor.ActorCell.invoke(ActorCell.scala:561)
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> akka.dispatch.Mailbox.run(Mailbox.scala:225)
> akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "}
> {"@timestamp":"2021-10-19T04:53:16.925+02:00","@version":"1","message":"Source: Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.926+02:00","@version":"1","message":"Source: Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.927+02:00","@version":"1","message":"Discarding the results produced by task execution 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.928+02:00","@version":"1","message":"Discarding the results produced by task execution 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding the results produced by task execution e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding the results produced by task execution e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding the results produced by task execution 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding the results produced by task execution 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Filter with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding the results produced by task execution 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding the results produced by task execution 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding the results produced by task execution 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding the results produced by task execution 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding the results produced by task execution 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding the results produced by task execution 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding the results produced by task execution f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding the results produced by task execution f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding the results produced by task execution a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding the results produced by task execution a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding the results produced by task execution 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding the results produced by task execution 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"httpAd call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding the results produced by task execution 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding the results produced by task execution 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding the results produced by task execution 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding the results produced by task execution 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding the results produced by task execution 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding the results produced by task execution 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Update Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Discarding the results produced by task execution 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.982+02:00","@version":"1","message":"Discarding the results produced by task execution 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink: Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink: Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.984+02:00","@version":"1","message":"Discarding the results produced by task execution 4467a95a23f6a5e983f006fb12832b10.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink: Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from RUNNING to CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink: Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from CANCELING to CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Discarding the results produced by task execution cf5ce6a12a584da4651a7d341cb1aeef.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Stopping checkpoint coordinator for job 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Shutting down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Job 00000000000000000000000000000000 has been suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Suspending SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Close ResourceManager connection 81a0f3b34590e11df305b58878a355b1: Stopping JobMaster for job flinkJob(00000000000000000000000000000000)..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Stopping SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Disconnect job manager bc901e3d25d406f70027b81673c742ba@akka.tcp://flink@172.24.10.160:6123/user/rpc/jobmanager_2 for job 00000000000000000000000000000000 from the resource manager.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Leader election started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Stopping DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Attempting to acquire leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@4d181edd","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5efd28bd","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Loop thread interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException: null
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Job 00000000000000000000000000000000 reached terminal state SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.996+02:00","@version":"1","message":"Released job graph 00000000000000000000000000000000 from KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-4","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.997+02:00","@version":"1","message":"Application failed unexpectedly: ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-4","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException: null
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:16.998+02:00","@version":"1","message":"Fatal error occurred in the cluster entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-4","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException: Application failed unexpectedly.
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.CancellationException: null
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
> \t... 22 common frames omitted
> "}
> {"@timestamp":"2021-10-19T04:53:17.084+02:00","@version":"1","message":"Shutting StandaloneApplicationClusterEntryPoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint shutdown hook","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.085+02:00","@version":"1","message":"Shutting down rest endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint shutdown hook","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.086+02:00","@version":"1","message":"Stopped BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer shutdown hook","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Successfully Acquired leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader (8f7dddbb-236a-43fe-a706-1795b494838d)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-25081-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Attempting to renew leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.185+02:00","@version":"1","message":"Removing cache directory /tmp/flink-web-682b3e4f-ed72-4b00-96a0-da5c8248b62c/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Stopping DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6f9e08d4","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@42c1c89c","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Loop thread interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException: null
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Shut down complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@11787b64","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@4c063eb4","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@4839a3ec","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@202d9236","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@d3b348f","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Loop thread interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException: null
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Stopping SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Closing the SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Suspending the SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Stopping DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7be008e9","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Loop thread interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException: null
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@57ae1976","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3f6e134d","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"The watcher is closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Closing websocket org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@2a1892aa","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"WebSocket close received. code: 1000, reason: ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring onClose for already closed/closing websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>
>