You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by mejri houssem <me...@gmail.com> on 2021/12/12 11:24:50 UTC
Re: Flink application mode using k8S HA.
Any updates on this subject please?
Le mar. 30 nov. 2021 à 20:55, marco andreas <ma...@gmail.com> a
écrit :
> Has anyone encountered the same problem and found a solution?
>
> Any suggestions would be appreciated.
>
> Thanks.
>
> Le lun. 29 nov. 2021 à 18:13, marco andreas <ma...@gmail.com> a
> écrit :
>
>> I am using the 1.12.5 flink version.
>>
>> Le lun. 29 nov. 2021 à 10:41, marco andreas <ma...@gmail.com>
>> a écrit :
>>
>>> Hello flink community,
>>>
>>> I am deploying a flink job on a standalone application cluster mode
>>> using kubernetes high availability.
>>>
>>> The job starts running normally but at some point in time it starts to
>>> crash and eventually after 5 restart the jobmanager pod gets deleted.
>>>
>>> Ps: I don't have any memory problems.
>>>
>>> Here's the log of the crashing job.
>>>
>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>> job1-flink - job1-restserver-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>> http://172.24.10.171:8081 lost
>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>> election
>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>> http://172.24.10.171:8081 was granted leadership with
>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>> job1-flink - job1-dispatcher-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>> dispatcher akka.tcp://flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>> all currently running jobs of dispatcher akka.tcp://
>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>> closing the watch
>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>> watcher is
>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>> websocket
>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>> close received. code: 1000, reason:
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>> onClose for already closed/closing
>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>> JobManager is no longer the leader.
>>>
>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>
>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>
>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>
>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>
>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>
>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>
>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>
>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>
>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>
>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>
>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run(Thread.java:748)"}
>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>> election
>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>> election
>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>> job1-00000000000000000000000000000000-jobmanager-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>> job1-flink - job1-resourcemanager-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>> revoked leadership. Clearing fencing
>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>> closing the watch
>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>> watcher is
>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>> to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>> websocket
>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>> the
>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>> close received. code: 1000, reason:
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>> onClose for already closed/closing
>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>> to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>> to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>> job1-00000000000000000000000000000000-jobmanager-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>> job1-00000000000000000000000000000000-jobmanager-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>> to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>> to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>> RUNNING to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>> CANCELING to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>> RUNNING to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>> CANCELING to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>> to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>> election
>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>> job1-resourcemanager-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>> to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>> RUNNING to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>> CANCELING to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>> RUNNING to
>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>> occurred while renewing lock: Unable to update
>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>> Unable to update ConfigMapLock
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run(Thread.java:748)
>>> Caused by: io.fabric8.kubernetes.client.KubernetesClientException:
>>> Failure executing: PUT at:
>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>> Message: Operation cannot be fulfilled on configmaps
>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>> your changes to the latest version and try again. Received status:
>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>> message=Operation cannot be fulfilled on configmaps
>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>> your changes to the latest version and try again,
>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>
>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>
>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>
>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>
>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>
>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>
>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>
>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>
>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>
>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>
>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>
>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>
>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>> \t... 10 common frames omitted"}
>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>> CANCELING to
>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>> the results produced by task execution
>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>> checkpoint coordinator for job
>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>
>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>> 00000000000000000000000000000000 has been
>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>> no longer the
>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>> the JobMaster for job
>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>> job1-resourcemanager-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>> job1-resourcemanager-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>> closing the watch
>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>> watcher is
>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>> thread
>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>> null
>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run(Thread.java:748)"}
>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>> websocket
>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>> close received. code: 1000, reason:
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>> onClose for already closed/closing
>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>> 00000000000000000000000000000000 reached terminal state
>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>> granted leadership with fencing token
>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>> the
>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>> collision: Group already contains a Metric with the name
>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>> Metric will not be reported.[172.24.10.171,
>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>> job graph 00000000000000000000000000000000 from
>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>> failed unexpectedly:
>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>> null
>>>
>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>
>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>
>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>
>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>
>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>
>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>
>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>
>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>
>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>
>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>
>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>
>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>
>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>
>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>
>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>
>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>
>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>
>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>
>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>
>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run(Thread.java:748)"}
>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>> error occurred in the cluster
>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>> Application failed unexpectedly.
>>>
>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>
>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>
>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>
>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>
>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>
>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>
>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>
>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>
>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>
>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>
>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>
>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>
>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>
>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>
>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>
>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>
>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>
>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>
>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>
>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>
>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>
>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>
>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>
>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run(Thread.java:748)
>>> Caused by: java.util.concurrent.CancellationException: null
>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>> 22 common frames omitted"}
>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>> StandaloneApplicationClusterEntryPoint down with application status
>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>> shutdown hook","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>> shutdown hook","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>> down rest
>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>> shutdown hook","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>> cache directory
>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>> closing the watch
>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>> watcher is
>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>> thread
>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>> null
>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run(Thread.java:748)"}
>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>> websocket
>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>> down
>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>> close received. code: 1000, reason:
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>> onClose for already closed/closing
>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>> closing the watch
>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>> watcher is
>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>> websocket
>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>> close received. code: 1000, reason:
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>> onClose for already closed/closing
>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>> closing the watch
>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>> watcher is
>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>> websocket
>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>> close received. code: 1000, reason:
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>> onClose for already closed/closing
>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>> closing the watch
>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>> watcher is
>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>> thread
>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>> null
>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run(Thread.java:748)"}
>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>> websocket
>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>> close received. code: 1000, reason:
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>> onClose for already closed/closing
>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>> the
>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>> the
>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>> closing the watch
>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>> thread
>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>> null
>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>
>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run(Thread.java:748)"}
>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>> watcher is
>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>> websocket
>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>> close received. code: 1000, reason:
>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>> onClose for already closed/closing
>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>
>>>
>>>
>>>
>>>
Re: Flink application mode using k8S HA.
Posted by Matthias Pohl <ma...@ververica.com>.
Marco sent the logs to me individually. Please find them attached to this
message. There are some permission issues when accessing the Flink
configuration in the docker-entrypoint.sh.
Additionally, the Hadoop dependency does not seem to be located on the
classpath: java.lang.NoClassDefFoundError:
org/apache/hadoop/hdfs/HdfsConfiguration. But these errors should be
unrelated to accessing the k8s ConfigMap.
On Mon, Dec 13, 2021 at 1:28 PM David Morávek <dm...@apache.org> wrote:
> Hi Marcos, can you please attach full log from the affected JM? The leader
> election settings you have seem quite generous.
>
> Best,
> D.
>
> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <ma...@gmail.com>
> wrote:
>
>> Hello David,
>>
>> So I am using the official flink setup to deploy a
>> standalone application cluster.
>>
>> For the k8S HA related setting I just changed the lease-period, you will
>> find attached the config-map settings.
>>
>> The reason is that the JM is not able to renew the lease period ,this
>> error is totally random , the job may run for hours without any crashes and
>> sometimes it starts to crash every 15mn or 30 mn ...
>>
>> Best,
>>
>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a écrit :
>>
>>> Hi Marco and Mejri,
>>>
>>> the email didn't get it pass the gmail's spam filter (fairly long
>>> json-based log?), which is most likely the reason no-one has picked it up.
>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>
>>> What I find most suspicious is this part:
>>>
>>> Renew deadline reached after 200 seconds while renewing lock
>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>
>>> 200 seconds seems like a quite some time for not being able to able to
>>> update the config-map with a new lease. Did you change any other k8s /
>>> leader election related settings?
>>>
>>> As for why JM might be gone, in application there is a known bug, that
>>> results in the hard-kill when leadership is lost [1].
>>>
>>> I'm also aware of other issues with the application mode, that has been
>>> fixed in 1.13.x and 1.14.x. Also there have been some significant
>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>> any chance you can try with the newer version of Flink?
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>
>>>
>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <me...@gmail.com>
>>> wrote:
>>>
>>>> Any updates on this subject please?
>>>>
>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <ma...@gmail.com>
>>>> a écrit :
>>>>
>>>>> Has anyone encountered the same problem and found a solution?
>>>>>
>>>>> Any suggestions would be appreciated.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <ma...@gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> I am using the 1.12.5 flink version.
>>>>>>
>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>
>>>>>>> Hello flink community,
>>>>>>>
>>>>>>> I am deploying a flink job on a standalone application cluster mode
>>>>>>> using kubernetes high availability.
>>>>>>>
>>>>>>> The job starts running normally but at some point in time it starts
>>>>>>> to crash and eventually after 5 restart the jobmanager pod gets deleted.
>>>>>>>
>>>>>>> Ps: I don't have any memory problems.
>>>>>>>
>>>>>>> Here's the log of the crashing job.
>>>>>>>
>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>> job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>> http://172.24.10.171:8081 lost
>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>> election
>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>> dispatcher akka.tcp://flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>> JobManager is no longer the leader.
>>>>>>>
>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>
>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>
>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>
>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>
>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>
>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>> election
>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>> election
>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>> revoked leadership. Clearing fencing
>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>> the
>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>> to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>> to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>> RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>> CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>> RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>> CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>> election
>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-resourcemanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>> RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>> CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>> RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>> occurred while renewing lock: Unable to update
>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>> Unable to update ConfigMapLock
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: io.fabric8.kubernetes.client.KubernetesClientException:
>>>>>>> Failure executing: PUT at:
>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>> your changes to the latest version and try again,
>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>> \t... 10 common frames omitted"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>> CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>> checkpoint coordinator for job
>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>> 00000000000000000000000000000000 has been
>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>> no longer the
>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>> the JobMaster for job
>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-resourcemanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-resourcemanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>> thread
>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>> granted leadership with fencing token
>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>> the
>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>> collision: Group already contains a Metric with the name
>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>> failed unexpectedly:
>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>
>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>
>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>> error occurred in the cluster
>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>> Application failed unexpectedly.
>>>>>>>
>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>
>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>
>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>> 22 common frames omitted"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>> down rest
>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>> cache directory
>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>> thread
>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>> down
>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>> thread
>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>> the
>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>> the
>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>> thread
>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>
>>>>>>
Re: Flink application mode using k8S HA.
Posted by Yang Wang <da...@gmail.com>.
>
> What I don't understand is why the application gets shut down when the api
> server is not responding.
The root cause is the Flink JobManager/ResourceManager lost the leadership
and then will restart the JobManager process.
If you could reproduce the issue in a minikube environment, then it will be
easier to debug.
Refer to [1] for how to enable the APIServer audit log for minikube.
[1]. https://minikube.sigs.k8s.io/docs/tutorials/audit-policy/
Best,
Yang
marco andreas <ma...@gmail.com> 于2022年1月28日周五 01:20写道：
> Hello ,
>
> With respect to the memory issue , I have no problem.
>
> Regarding the k8S APISERVER, I can't check the log because it's
> a mutualised cluster and I don't have the rights to do so .
>
> What I don't understand is why the application gets shut down when the api
> server is not responding.
>
> In the meantime do you have any solutions to get around this problem by
> the time I can get the api server log ?
>
> Thanks,
>
>
>
> Le mar. 11 janv. 2022 à 04:11, Yang Wang <da...@gmail.com> a écrit :
>
>> You could add the following config option to enable GC log.
>>
>> env.java.opts.jobmanager: -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/opt/flink/log/jobmanager-gc.log
>>
>> Given that your JobManager heap is not too large(> 20Gb), the 300s fullGC
>> should not happen. But we could enable the GC log for double checking.
>>
>>
>> Another suspicion is that K8s APIServer might be overloaded. You need to
>> check the K8s APIServer audit log to confirm whether the renew ConfigMap
>> request has been received.
>>
>>
>> Best,
>> Yang
>>
>>
>> marco andreas <ma...@gmail.com> 于2022年1月11日周二 02:22写道：
>>
>>> Hello Yang,
>>>
>>> Yes, that's the problem. The jobmanager wast'n able to renew the lease
>>> period despite the 300s of time I had configured.
>>>
>>> Could you please tell how to verify how I can tell that the JobManager
>>> pod was having a long fullGC.
>>>
>>> Best,
>>>
>>> Le lun. 10 janv. 2022 à 04:42, Yang Wang <da...@gmail.com> a
>>> écrit :
>>>
>>>> The job failed because of "Renew deadline reached after 300 seconds
>>>> while renewing lock".
>>>>
>>>> Could you please verify that the JobManager pod was having a long
>>>> fullGC or something is wrong with the network from 2022-01-08T15:34:22 to
>>>> 2022-01-08T15:37:07?
>>>>
>>>>
>>>>
>>>>
>>>> Best,
>>>> Yang
>>>>
>>>> marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道：
>>>>
>>>>> Hello,
>>>>>
>>>>> I upgraded the flink version lately to 1.13.5, however i
>>>>> encountered the same problem.
>>>>>
>>>>> Any thoughts on this issue ?
>>>>>
>>>>> JM Log in the attached file.
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> I am sure that the kubernetes.cluster-id is not being reused (it's
>>>>>> unique ), and I have only one instance of the jobmanager in my cluster.
>>>>>>
>>>>>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a
>>>>>> écrit :
>>>>>>
>>>>>>> Is it possible that you might be reusing the same
>>>>>>> kubernetes.cluster-id (job1) between multiple flink clusters? How many
>>>>>>> jobmanagers are you running for the cluster?
>>>>>>>
>>>>>>> D.
>>>>>>>
>>>>>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <
>>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi David,
>>>>>>>>
>>>>>>>> I dont have the full log.
>>>>>>>> But here's the full log of a crashed JM with similar configurations
>>>>>>>> (just the lease period is different : 60 s )
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>>
>>>>>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> Hi Marcos, can you please attach full log from the affected JM?
>>>>>>>>> The leader election settings you have seem quite generous.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello David,
>>>>>>>>>>
>>>>>>>>>> So I am using the official flink setup to deploy a
>>>>>>>>>> standalone application cluster.
>>>>>>>>>>
>>>>>>>>>> For the k8S HA related setting I just changed the lease-period,
>>>>>>>>>> you will find attached the config-map settings.
>>>>>>>>>>
>>>>>>>>>> The reason is that the JM is not able to renew the lease
>>>>>>>>>> period ,this error is totally random , the job may run for hours without
>>>>>>>>>> any crashes and sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>>>>>> écrit :
>>>>>>>>>>
>>>>>>>>>>> Hi Marco and Mejri,
>>>>>>>>>>>
>>>>>>>>>>> the email didn't get it pass the gmail's spam filter (fairly
>>>>>>>>>>> long json-based log?), which is most likely the reason no-one has picked it
>>>>>>>>>>> up. Can you please elaborate bit more about your setup (scale, k8s cluster,
>>>>>>>>>>> ...)
>>>>>>>>>>>
>>>>>>>>>>> What I find most suspicious is this part:
>>>>>>>>>>>
>>>>>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>>>>>
>>>>>>>>>>> 200 seconds seems like a quite some time for not being able to
>>>>>>>>>>> able to update the config-map with a new lease. Did you change any other
>>>>>>>>>>> k8s / leader election related settings?
>>>>>>>>>>>
>>>>>>>>>>> As for why JM might be gone, in application there is a known
>>>>>>>>>>> bug, that results in the hard-kill when leadership is lost [1].
>>>>>>>>>>>
>>>>>>>>>>> I'm also aware of other issues with the application mode, that
>>>>>>>>>>> has been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>>>>>
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Any updates on this subject please?
>>>>>>>>>>>>
>>>>>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am deploying a flink job on a standalone application
>>>>>>>>>>>>>>> cluster mode using kubernetes high availability.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>>>>>> deleted.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>>> election
>>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>>> election
>>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>>> election
>>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>>>> akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>>>>>>>>> revoked leadership. Clearing fencing
>>>>>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>>> election
>>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>>>>>> PUT at:
>>>>>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>>>>>> no longer the
>>>>>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>>>> akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>>>>>>>>> granted leadership with fencing token
>>>>>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>>> down rest
>>>>>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>>>>>> cache directory
>>>>>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>>>>>> down
>>>>>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
Re: Flink application mode using k8S HA.
Posted by marco andreas <ma...@gmail.com>.
Hello ,
With respect to the memory issue , I have no problem.
Regarding the k8S APISERVER, I can't check the log because it's
a mutualised cluster and I don't have the rights to do so .
What I don't understand is why the application gets shut down when the api
server is not responding.
In the meantime do you have any solutions to get around this problem by
the time I can get the api server log ?
Thanks,
Le mar. 11 janv. 2022 à 04:11, Yang Wang <da...@gmail.com> a écrit :
> You could add the following config option to enable GC log.
>
> env.java.opts.jobmanager: -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/opt/flink/log/jobmanager-gc.log
>
> Given that your JobManager heap is not too large(> 20Gb), the 300s fullGC
> should not happen. But we could enable the GC log for double checking.
>
>
> Another suspicion is that K8s APIServer might be overloaded. You need to
> check the K8s APIServer audit log to confirm whether the renew ConfigMap
> request has been received.
>
>
> Best,
> Yang
>
>
> marco andreas <ma...@gmail.com> 于2022年1月11日周二 02:22写道：
>
>> Hello Yang,
>>
>> Yes, that's the problem. The jobmanager wast'n able to renew the lease
>> period despite the 300s of time I had configured.
>>
>> Could you please tell how to verify how I can tell that the JobManager
>> pod was having a long fullGC.
>>
>> Best,
>>
>> Le lun. 10 janv. 2022 à 04:42, Yang Wang <da...@gmail.com> a
>> écrit :
>>
>>> The job failed because of "Renew deadline reached after 300 seconds
>>> while renewing lock".
>>>
>>> Could you please verify that the JobManager pod was having a long fullGC
>>> or something is wrong with the network from 2022-01-08T15:34:22 to
>>> 2022-01-08T15:37:07?
>>>
>>>
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道：
>>>
>>>> Hello,
>>>>
>>>> I upgraded the flink version lately to 1.13.5, however i
>>>> encountered the same problem.
>>>>
>>>> Any thoughts on this issue ?
>>>>
>>>> JM Log in the attached file.
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com>
>>>> a écrit :
>>>>
>>>>> I am sure that the kubernetes.cluster-id is not being reused (it's
>>>>> unique ), and I have only one instance of the jobmanager in my cluster.
>>>>>
>>>>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a
>>>>> écrit :
>>>>>
>>>>>> Is it possible that you might be reusing the same
>>>>>> kubernetes.cluster-id (job1) between multiple flink clusters? How many
>>>>>> jobmanagers are you running for the cluster?
>>>>>>
>>>>>> D.
>>>>>>
>>>>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <
>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>
>>>>>>> Hi David,
>>>>>>>
>>>>>>> I dont have the full log.
>>>>>>> But here's the full log of a crashed JM with similar configurations
>>>>>>> (just the lease period is different : 60 s )
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>>
>>>>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>>>>>> leader election settings you have seem quite generous.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> D.
>>>>>>>>
>>>>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello David,
>>>>>>>>>
>>>>>>>>> So I am using the official flink setup to deploy a
>>>>>>>>> standalone application cluster.
>>>>>>>>>
>>>>>>>>> For the k8S HA related setting I just changed the lease-period,
>>>>>>>>> you will find attached the config-map settings.
>>>>>>>>>
>>>>>>>>> The reason is that the JM is not able to renew the lease
>>>>>>>>> period ,this error is totally random , the job may run for hours without
>>>>>>>>> any crashes and sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>>> Hi Marco and Mejri,
>>>>>>>>>>
>>>>>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>>>>>
>>>>>>>>>> What I find most suspicious is this part:
>>>>>>>>>>
>>>>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>>>>
>>>>>>>>>> 200 seconds seems like a quite some time for not being able to
>>>>>>>>>> able to update the config-map with a new lease. Did you change any other
>>>>>>>>>> k8s / leader election related settings?
>>>>>>>>>>
>>>>>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>>>>>
>>>>>>>>>> I'm also aware of other issues with the application mode, that
>>>>>>>>>> has been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Any updates on this subject please?
>>>>>>>>>>>
>>>>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>>>>
>>>>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am deploying a flink job on a standalone application
>>>>>>>>>>>>>> cluster mode using kubernetes high availability.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>>>>> deleted.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>> election
>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>> election
>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>> election
>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>>> akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>>>>>>>> revoked leadership. Clearing fencing
>>>>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>> election
>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>>>>> PUT at:
>>>>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>>>>> no longer the
>>>>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>>> akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>>>>>>>> granted leadership with fencing token
>>>>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>> down rest
>>>>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>>>>> cache directory
>>>>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>>>>> down
>>>>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
Re: Flink application mode using k8S HA.
Posted by Yang Wang <da...@gmail.com>.
You could add the following config option to enable GC log.
env.java.opts.jobmanager: -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/opt/flink/log/jobmanager-gc.log
Given that your JobManager heap is not too large(> 20Gb), the 300s fullGC
should not happen. But we could enable the GC log for double checking.
Another suspicion is that K8s APIServer might be overloaded. You need to
check the K8s APIServer audit log to confirm whether the renew ConfigMap
request has been received.
Best,
Yang
marco andreas <ma...@gmail.com> 于2022年1月11日周二 02:22写道：
> Hello Yang,
>
> Yes, that's the problem. The jobmanager wast'n able to renew the lease
> period despite the 300s of time I had configured.
>
> Could you please tell how to verify how I can tell that the JobManager pod
> was having a long fullGC.
>
> Best,
>
> Le lun. 10 janv. 2022 à 04:42, Yang Wang <da...@gmail.com> a écrit :
>
>> The job failed because of "Renew deadline reached after 300 seconds while
>> renewing lock".
>>
>> Could you please verify that the JobManager pod was having a long fullGC
>> or something is wrong with the network from 2022-01-08T15:34:22 to
>> 2022-01-08T15:37:07?
>>
>>
>>
>>
>> Best,
>> Yang
>>
>> marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道：
>>
>>> Hello,
>>>
>>> I upgraded the flink version lately to 1.13.5, however i encountered the
>>> same problem.
>>>
>>> Any thoughts on this issue ?
>>>
>>> JM Log in the attached file.
>>>
>>> Best regards,
>>>
>>>
>>> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com>
>>> a écrit :
>>>
>>>> I am sure that the kubernetes.cluster-id is not being reused (it's
>>>> unique ), and I have only one instance of the jobmanager in my cluster.
>>>>
>>>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a écrit :
>>>>
>>>>> Is it possible that you might be reusing the same
>>>>> kubernetes.cluster-id (job1) between multiple flink clusters? How many
>>>>> jobmanagers are you running for the cluster?
>>>>>
>>>>> D.
>>>>>
>>>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <
>>>>> marcoandreas582@gmail.com> wrote:
>>>>>
>>>>>> Hi David,
>>>>>>
>>>>>> I dont have the full log.
>>>>>> But here's the full log of a crashed JM with similar configurations
>>>>>> (just the lease period is different : 60 s )
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>>
>>>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a
>>>>>> écrit :
>>>>>>
>>>>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>>>>> leader election settings you have seem quite generous.
>>>>>>>
>>>>>>> Best,
>>>>>>> D.
>>>>>>>
>>>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello David,
>>>>>>>>
>>>>>>>> So I am using the official flink setup to deploy a
>>>>>>>> standalone application cluster.
>>>>>>>>
>>>>>>>> For the k8S HA related setting I just changed the lease-period, you
>>>>>>>> will find attached the config-map settings.
>>>>>>>>
>>>>>>>> The reason is that the JM is not able to renew the lease
>>>>>>>> period ,this error is totally random , the job may run for hours without
>>>>>>>> any crashes and sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> Hi Marco and Mejri,
>>>>>>>>>
>>>>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>>>>
>>>>>>>>> What I find most suspicious is this part:
>>>>>>>>>
>>>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>>>
>>>>>>>>> 200 seconds seems like a quite some time for not being able to
>>>>>>>>> able to update the config-map with a new lease. Did you change any other
>>>>>>>>> k8s / leader election related settings?
>>>>>>>>>
>>>>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>>>>
>>>>>>>>> I'm also aware of other issues with the application mode, that has
>>>>>>>>> been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>>>
>>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Any updates on this subject please?
>>>>>>>>>>
>>>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>>>
>>>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>>>
>>>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am deploying a flink job on a standalone application
>>>>>>>>>>>>> cluster mode using kubernetes high availability.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>>>> deleted.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>>>
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>> election
>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>> election
>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>> election
>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>>>> was revoked leadership. Clearing fencing
>>>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>> election
>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>>>> PUT at:
>>>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>>>> no longer the
>>>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>> thread
>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>>>> was granted leadership with fencing token
>>>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>> down rest
>>>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>>>> cache directory
>>>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>> thread
>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>>>> down
>>>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>> thread
>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>> thread
>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>
>>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
Re: Flink application mode using k8S HA.
Posted by marco andreas <ma...@gmail.com>.
Hello Yang,
Yes, that's the problem. The jobmanager wast'n able to renew the lease
period despite the 300s of time I had configured.
Could you please tell how to verify how I can tell that the JobManager pod
was having a long fullGC.
Best,
Le lun. 10 janv. 2022 à 04:42, Yang Wang <da...@gmail.com> a écrit :
> The job failed because of "Renew deadline reached after 300 seconds while
> renewing lock".
>
> Could you please verify that the JobManager pod was having a long fullGC
> or something is wrong with the network from 2022-01-08T15:34:22 to
> 2022-01-08T15:37:07?
>
>
>
>
> Best,
> Yang
>
> marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道：
>
>> Hello,
>>
>> I upgraded the flink version lately to 1.13.5, however i encountered the
>> same problem.
>>
>> Any thoughts on this issue ?
>>
>> JM Log in the attached file.
>>
>> Best regards,
>>
>>
>> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com>
>> a écrit :
>>
>>> I am sure that the kubernetes.cluster-id is not being reused (it's
>>> unique ), and I have only one instance of the jobmanager in my cluster.
>>>
>>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a écrit :
>>>
>>>> Is it possible that you might be reusing the same kubernetes.cluster-id
>>>> (job1) between multiple flink clusters? How many jobmanagers are you
>>>> running for the cluster?
>>>>
>>>> D.
>>>>
>>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <
>>>> marcoandreas582@gmail.com> wrote:
>>>>
>>>>> Hi David,
>>>>>
>>>>> I dont have the full log.
>>>>> But here's the full log of a crashed JM with similar configurations
>>>>> (just the lease period is different : 60 s )
>>>>>
>>>>> Best,
>>>>>
>>>>>
>>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a
>>>>> écrit :
>>>>>
>>>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>>>> leader election settings you have seem quite generous.
>>>>>>
>>>>>> Best,
>>>>>> D.
>>>>>>
>>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>
>>>>>>> Hello David,
>>>>>>>
>>>>>>> So I am using the official flink setup to deploy a
>>>>>>> standalone application cluster.
>>>>>>>
>>>>>>> For the k8S HA related setting I just changed the lease-period, you
>>>>>>> will find attached the config-map settings.
>>>>>>>
>>>>>>> The reason is that the JM is not able to renew the lease
>>>>>>> period ,this error is totally random , the job may run for hours without
>>>>>>> any crashes and sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Hi Marco and Mejri,
>>>>>>>>
>>>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>>>
>>>>>>>> What I find most suspicious is this part:
>>>>>>>>
>>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>>
>>>>>>>> 200 seconds seems like a quite some time for not being able to able
>>>>>>>> to update the config-map with a new lease. Did you change any other k8s /
>>>>>>>> leader election related settings?
>>>>>>>>
>>>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>>>
>>>>>>>> I'm also aware of other issues with the application mode, that has
>>>>>>>> been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Any updates on this subject please?
>>>>>>>>>
>>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>>
>>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>>
>>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>>
>>>>>>>>>>>> I am deploying a flink job on a standalone application cluster
>>>>>>>>>>>> mode using kubernetes high availability.
>>>>>>>>>>>>
>>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>>> deleted.
>>>>>>>>>>>>
>>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>>
>>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>>
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>>> election
>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>>> election
>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>>> election
>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>>> was revoked leadership. Clearing fencing
>>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>> the
>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>>> election
>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>> Caused by:
>>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>>> PUT at:
>>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>>> no longer the
>>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>>> thread
>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>>> was granted leadership with fencing token
>>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>>> the
>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>> down rest
>>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>>> cache directory
>>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>>> thread
>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>>> down
>>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>>> thread
>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> the
>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>> the
>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>>> thread
>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>
>>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
Re: Flink application mode using k8S HA.
Posted by Yang Wang <da...@gmail.com>.
The job failed because of "Renew deadline reached after 300 seconds while
renewing lock".
Could you please verify that the JobManager pod was having a long fullGC or
something is wrong with the network from 2022-01-08T15:34:22 to
2022-01-08T15:37:07?
Best,
Yang
marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道：
> Hello,
>
> I upgraded the flink version lately to 1.13.5, however i encountered the
> same problem.
>
> Any thoughts on this issue ?
>
> JM Log in the attached file.
>
> Best regards,
>
>
> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com> a
> écrit :
>
>> I am sure that the kubernetes.cluster-id is not being reused (it's unique
>> ), and I have only one instance of the jobmanager in my cluster.
>>
>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a écrit :
>>
>>> Is it possible that you might be reusing the same kubernetes.cluster-id
>>> (job1) between multiple flink clusters? How many jobmanagers are you
>>> running for the cluster?
>>>
>>> D.
>>>
>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <ma...@gmail.com>
>>> wrote:
>>>
>>>> Hi David,
>>>>
>>>> I dont have the full log.
>>>> But here's the full log of a crashed JM with similar configurations
>>>> (just the lease period is different : 60 s )
>>>>
>>>> Best,
>>>>
>>>>
>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a écrit :
>>>>
>>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>>> leader election settings you have seem quite generous.
>>>>>
>>>>> Best,
>>>>> D.
>>>>>
>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>> marcoandreas582@gmail.com> wrote:
>>>>>
>>>>>> Hello David,
>>>>>>
>>>>>> So I am using the official flink setup to deploy a
>>>>>> standalone application cluster.
>>>>>>
>>>>>> For the k8S HA related setting I just changed the lease-period, you
>>>>>> will find attached the config-map settings.
>>>>>>
>>>>>> The reason is that the JM is not able to renew the lease period ,this
>>>>>> error is totally random , the job may run for hours without any crashes and
>>>>>> sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>> écrit :
>>>>>>
>>>>>>> Hi Marco and Mejri,
>>>>>>>
>>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>>
>>>>>>> What I find most suspicious is this part:
>>>>>>>
>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>
>>>>>>> 200 seconds seems like a quite some time for not being able to able
>>>>>>> to update the config-map with a new lease. Did you change any other k8s /
>>>>>>> leader election related settings?
>>>>>>>
>>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>>
>>>>>>> I'm also aware of other issues with the application mode, that has
>>>>>>> been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>
>>>>>>>> Any updates on this subject please?
>>>>>>>>
>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>
>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>
>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>
>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>
>>>>>>>>>>> I am deploying a flink job on a standalone application cluster
>>>>>>>>>>> mode using kubernetes high availability.
>>>>>>>>>>>
>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>> deleted.
>>>>>>>>>>>
>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>
>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>> election
>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>> election
>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>> election
>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>> was revoked leadership. Clearing fencing
>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>> the
>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>> RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>> CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>> RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>> CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>> election
>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>> RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>> CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>> RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>> Caused by:
>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>> PUT at:
>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>> CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>> no longer the
>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>> thread
>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>> was granted leadership with fencing token
>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>> the
>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>> down rest
>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>> cache directory
>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>> thread
>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>> down
>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>> thread
>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>> the
>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>> the
>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>> thread
>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>
>>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
Re: Flink application mode using k8S HA.
Posted by marco andreas <ma...@gmail.com>.
Hello,
I upgraded the flink version lately to 1.13.5, however i encountered the
same problem.
Any thoughts on this issue ?
JM Log in the attached file.
Best regards,
Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com> a
écrit :
> I am sure that the kubernetes.cluster-id is not being reused (it's unique
> ), and I have only one instance of the jobmanager in my cluster.
>
> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a écrit :
>
>> Is it possible that you might be reusing the same kubernetes.cluster-id
>> (job1) between multiple flink clusters? How many jobmanagers are you
>> running for the cluster?
>>
>> D.
>>
>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <ma...@gmail.com>
>> wrote:
>>
>>> Hi David,
>>>
>>> I dont have the full log.
>>> But here's the full log of a crashed JM with similar configurations
>>> (just the lease period is different : 60 s )
>>>
>>> Best,
>>>
>>>
>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a écrit :
>>>
>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>> leader election settings you have seem quite generous.
>>>>
>>>> Best,
>>>> D.
>>>>
>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>> marcoandreas582@gmail.com> wrote:
>>>>
>>>>> Hello David,
>>>>>
>>>>> So I am using the official flink setup to deploy a
>>>>> standalone application cluster.
>>>>>
>>>>> For the k8S HA related setting I just changed the lease-period, you
>>>>> will find attached the config-map settings.
>>>>>
>>>>> The reason is that the JM is not able to renew the lease period ,this
>>>>> error is totally random , the job may run for hours without any crashes and
>>>>> sometimes it starts to crash every 15mn or 30 mn ...
>>>>>
>>>>> Best,
>>>>>
>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>> écrit :
>>>>>
>>>>>> Hi Marco and Mejri,
>>>>>>
>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>
>>>>>> What I find most suspicious is this part:
>>>>>>
>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>
>>>>>> 200 seconds seems like a quite some time for not being able to able
>>>>>> to update the config-map with a new lease. Did you change any other k8s /
>>>>>> leader election related settings?
>>>>>>
>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>
>>>>>> I'm also aware of other issues with the application mode, that has
>>>>>> been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>> any chance you can try with the newer version of Flink?
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>
>>>>>>> Any updates on this subject please?
>>>>>>>
>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>
>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>
>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>
>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>
>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>>> Hello flink community,
>>>>>>>>>>
>>>>>>>>>> I am deploying a flink job on a standalone application cluster
>>>>>>>>>> mode using kubernetes high availability.
>>>>>>>>>>
>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>> deleted.
>>>>>>>>>>
>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>
>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>
>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>> election
>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>> election
>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>> election
>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>> was revoked leadership. Clearing fencing
>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>> the
>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>> to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>> to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>> RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>> CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>> RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>> CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>> election
>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>> RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>> CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>> RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>> Caused by:
>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>> PUT at:
>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>> CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>> no longer the
>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>> the JobMaster for job
>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>> thread
>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>> was granted leadership with fencing token
>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>> the
>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>> failed unexpectedly:
>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>> error occurred in the cluster
>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>> down rest
>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>> cache directory
>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>> thread
>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>> down
>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>> thread
>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>> the
>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>> the
>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>> thread
>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>
>>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
Re: Flink application mode using k8S HA.
Posted by marco andreas <ma...@gmail.com>.
I am sure that the kubernetes.cluster-id is not being reused (it's unique
), and I have only one instance of the jobmanager in my cluster.
Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a écrit :
> Is it possible that you might be reusing the same kubernetes.cluster-id
> (job1) between multiple flink clusters? How many jobmanagers are you
> running for the cluster?
>
> D.
>
> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <ma...@gmail.com>
> wrote:
>
>> Hi David,
>>
>> I dont have the full log.
>> But here's the full log of a crashed JM with similar configurations (just
>> the lease period is different : 60 s )
>>
>> Best,
>>
>>
>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a écrit :
>>
>>> Hi Marcos, can you please attach full log from the affected JM? The
>>> leader election settings you have seem quite generous.
>>>
>>> Best,
>>> D.
>>>
>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>> marcoandreas582@gmail.com> wrote:
>>>
>>>> Hello David,
>>>>
>>>> So I am using the official flink setup to deploy a
>>>> standalone application cluster.
>>>>
>>>> For the k8S HA related setting I just changed the lease-period, you
>>>> will find attached the config-map settings.
>>>>
>>>> The reason is that the JM is not able to renew the lease period ,this
>>>> error is totally random , the job may run for hours without any crashes and
>>>> sometimes it starts to crash every 15mn or 30 mn ...
>>>>
>>>> Best,
>>>>
>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a écrit :
>>>>
>>>>> Hi Marco and Mejri,
>>>>>
>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>
>>>>> What I find most suspicious is this part:
>>>>>
>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>
>>>>> 200 seconds seems like a quite some time for not being able to able to
>>>>> update the config-map with a new lease. Did you change any other k8s /
>>>>> leader election related settings?
>>>>>
>>>>> As for why JM might be gone, in application there is a known bug, that
>>>>> results in the hard-kill when leadership is lost [1].
>>>>>
>>>>> I'm also aware of other issues with the application mode, that has
>>>>> been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>> any chance you can try with the newer version of Flink?
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>
>>>>>
>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>
>>>>>> Any updates on this subject please?
>>>>>>
>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>
>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>
>>>>>>> Any suggestions would be appreciated.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>
>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>
>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>
>>>>>>>>> Hello flink community,
>>>>>>>>>
>>>>>>>>> I am deploying a flink job on a standalone application cluster
>>>>>>>>> mode using kubernetes high availability.
>>>>>>>>>
>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>> deleted.
>>>>>>>>>
>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>
>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>
>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>> election
>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>> closing the watch
>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>> watcher is
>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>> websocket
>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>> onClose for already closed/closing
>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>
>>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>> election
>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>> election
>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>> was revoked leadership. Clearing fencing
>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>> closing the watch
>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>> watcher is
>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>> to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>> websocket
>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>> the
>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>> onClose for already closed/closing
>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>> to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>> to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>> to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>> to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>> RUNNING to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>> CANCELING to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>> RUNNING to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>> CANCELING to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>> to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>> election
>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>> to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>> RUNNING to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>> CANCELING to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>> RUNNING to
>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>> Caused by: io.fabric8.kubernetes.client.KubernetesClientException:
>>>>>>>>> Failure executing: PUT at:
>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>> CANCELING to
>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>> the results produced by task execution
>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>> checkpoint coordinator for job
>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>> no longer the
>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>> the JobMaster for job
>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>> closing the watch
>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>> watcher is
>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>> thread
>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>> null
>>>>>>>>>
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>> websocket
>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>> onClose for already closed/closing
>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>> was granted leadership with fencing token
>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>> the
>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>> failed unexpectedly:
>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>> null
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>
>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>> error occurred in the cluster
>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>
>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>
>>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>
>>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>> 22 common frames omitted"}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>> down rest
>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>> cache directory
>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>> closing the watch
>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>> watcher is
>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>> thread
>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>> null
>>>>>>>>>
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>> websocket
>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>> down
>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>> onClose for already closed/closing
>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>> closing the watch
>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>> watcher is
>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>> websocket
>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>> onClose for already closed/closing
>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>> closing the watch
>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>> watcher is
>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>> websocket
>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>> onClose for already closed/closing
>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>> closing the watch
>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>> watcher is
>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>> thread
>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>> null
>>>>>>>>>
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>> websocket
>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>> onClose for already closed/closing
>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>> the
>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>> the
>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>> closing the watch
>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>> thread
>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>> null
>>>>>>>>>
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>
>>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>> watcher is
>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>> websocket
>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>> onClose for already closed/closing
>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
Re: Flink application mode using k8S HA.
Posted by David Morávek <dm...@apache.org>.
Is it possible that you might be reusing the same kubernetes.cluster-id
(job1) between multiple flink clusters? How many jobmanagers are you
running for the cluster?
D.
On Mon, Dec 13, 2021 at 2:37 PM marco andreas <ma...@gmail.com>
wrote:
> Hi David,
>
> I dont have the full log.
> But here's the full log of a crashed JM with similar configurations (just
> the lease period is different : 60 s )
>
> Best,
>
>
> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a écrit :
>
>> Hi Marcos, can you please attach full log from the affected JM? The
>> leader election settings you have seem quite generous.
>>
>> Best,
>> D.
>>
>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <ma...@gmail.com>
>> wrote:
>>
>>> Hello David,
>>>
>>> So I am using the official flink setup to deploy a
>>> standalone application cluster.
>>>
>>> For the k8S HA related setting I just changed the lease-period, you will
>>> find attached the config-map settings.
>>>
>>> The reason is that the JM is not able to renew the lease period ,this
>>> error is totally random , the job may run for hours without any crashes and
>>> sometimes it starts to crash every 15mn or 30 mn ...
>>>
>>> Best,
>>>
>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a écrit :
>>>
>>>> Hi Marco and Mejri,
>>>>
>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>
>>>> What I find most suspicious is this part:
>>>>
>>>> Renew deadline reached after 200 seconds while renewing lock
>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>
>>>> 200 seconds seems like a quite some time for not being able to able to
>>>> update the config-map with a new lease. Did you change any other k8s /
>>>> leader election related settings?
>>>>
>>>> As for why JM might be gone, in application there is a known bug, that
>>>> results in the hard-kill when leadership is lost [1].
>>>>
>>>> I'm also aware of other issues with the application mode, that has been
>>>> fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>> any chance you can try with the newer version of Flink?
>>>>
>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>
>>>>
>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>> mejrihoussem09@gmail.com> wrote:
>>>>
>>>>> Any updates on this subject please?
>>>>>
>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <ma...@gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>
>>>>>> Any suggestions would be appreciated.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>
>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>
>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>
>>>>>>>> Hello flink community,
>>>>>>>>
>>>>>>>> I am deploying a flink job on a standalone application cluster
>>>>>>>> mode using kubernetes high availability.
>>>>>>>>
>>>>>>>> The job starts running normally but at some point in time it starts
>>>>>>>> to crash and eventually after 5 restart the jobmanager pod gets deleted.
>>>>>>>>
>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>
>>>>>>>> Here's the log of the crashing job.
>>>>>>>>
>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>
>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>> election
>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>
>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>> dispatcher akka.tcp://
>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>> closing the watch
>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>> watcher is
>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>> websocket
>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>> close received. code: 1000, reason:
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>> onClose for already closed/closing
>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>> JobManager is no longer the leader.
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>
>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>> election
>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>> election
>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>> revoked leadership. Clearing fencing
>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>> closing the watch
>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>> watcher is
>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>> to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>> websocket
>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>> the
>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>> close received. code: 1000, reason:
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>> onClose for already closed/closing
>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>> to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>> to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>> to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>> to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>> RUNNING to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>> CANCELING to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>> RUNNING to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>> CANCELING to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>> to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>> election
>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>> job1-resourcemanager-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>> to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>> RUNNING to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>> CANCELING to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>> RUNNING to
>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>> Caused by: io.fabric8.kubernetes.client.KubernetesClientException:
>>>>>>>> Failure executing: PUT at:
>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>> your changes to the latest version and try again,
>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>> CANCELING to
>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>> the results produced by task execution
>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>> checkpoint coordinator for job
>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>> no longer the
>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>> the JobMaster for job
>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>> job1-resourcemanager-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>> job1-resourcemanager-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>> closing the watch
>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>> watcher is
>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>> thread
>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>> null
>>>>>>>>
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>> websocket
>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>> close received. code: 1000, reason:
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>> onClose for already closed/closing
>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>> granted leadership with fencing token
>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>> the
>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>> failed unexpectedly:
>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>> null
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>
>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>> error occurred in the cluster
>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>> Application failed unexpectedly.
>>>>>>>>
>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>
>>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>
>>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>
>>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>> 22 common frames omitted"}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>> down rest
>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>> cache directory
>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>> closing the watch
>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>> watcher is
>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>> thread
>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>> null
>>>>>>>>
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>> websocket
>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>> down
>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>> close received. code: 1000, reason:
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>> onClose for already closed/closing
>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>> closing the watch
>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>> watcher is
>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>> websocket
>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>> close received. code: 1000, reason:
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>> onClose for already closed/closing
>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>> closing the watch
>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>> watcher is
>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>> websocket
>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>> close received. code: 1000, reason:
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>> onClose for already closed/closing
>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>> closing the watch
>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>> watcher is
>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>> thread
>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>> null
>>>>>>>>
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>> websocket
>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>> close received. code: 1000, reason:
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>> onClose for already closed/closing
>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>> the
>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>> the
>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>> closing the watch
>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>> thread
>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>> null
>>>>>>>>
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>
>>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>> watcher is
>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>> websocket
>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>> close received. code: 1000, reason:
>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>> onClose for already closed/closing
>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
Re: Flink application mode using k8S HA.
Posted by marco andreas <ma...@gmail.com>.
Hi David,
I dont have the full log.
But here's the full log of a crashed JM with similar configurations (just
the lease period is different : 60 s )
Best,
Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a écrit :
> Hi Marcos, can you please attach full log from the affected JM? The leader
> election settings you have seem quite generous.
>
> Best,
> D.
>
> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <ma...@gmail.com>
> wrote:
>
>> Hello David,
>>
>> So I am using the official flink setup to deploy a
>> standalone application cluster.
>>
>> For the k8S HA related setting I just changed the lease-period, you will
>> find attached the config-map settings.
>>
>> The reason is that the JM is not able to renew the lease period ,this
>> error is totally random , the job may run for hours without any crashes and
>> sometimes it starts to crash every 15mn or 30 mn ...
>>
>> Best,
>>
>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a écrit :
>>
>>> Hi Marco and Mejri,
>>>
>>> the email didn't get it pass the gmail's spam filter (fairly long
>>> json-based log?), which is most likely the reason no-one has picked it up.
>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>
>>> What I find most suspicious is this part:
>>>
>>> Renew deadline reached after 200 seconds while renewing lock
>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>
>>> 200 seconds seems like a quite some time for not being able to able to
>>> update the config-map with a new lease. Did you change any other k8s /
>>> leader election related settings?
>>>
>>> As for why JM might be gone, in application there is a known bug, that
>>> results in the hard-kill when leadership is lost [1].
>>>
>>> I'm also aware of other issues with the application mode, that has been
>>> fixed in 1.13.x and 1.14.x. Also there have been some significant
>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>> any chance you can try with the newer version of Flink?
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>
>>>
>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <me...@gmail.com>
>>> wrote:
>>>
>>>> Any updates on this subject please?
>>>>
>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <ma...@gmail.com>
>>>> a écrit :
>>>>
>>>>> Has anyone encountered the same problem and found a solution?
>>>>>
>>>>> Any suggestions would be appreciated.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <ma...@gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> I am using the 1.12.5 flink version.
>>>>>>
>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>
>>>>>>> Hello flink community,
>>>>>>>
>>>>>>> I am deploying a flink job on a standalone application cluster mode
>>>>>>> using kubernetes high availability.
>>>>>>>
>>>>>>> The job starts running normally but at some point in time it starts
>>>>>>> to crash and eventually after 5 restart the jobmanager pod gets deleted.
>>>>>>>
>>>>>>> Ps: I don't have any memory problems.
>>>>>>>
>>>>>>> Here's the log of the crashing job.
>>>>>>>
>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>> job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>> http://172.24.10.171:8081 lost
>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>> election
>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>> dispatcher akka.tcp://flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>> JobManager is no longer the leader.
>>>>>>>
>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>
>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>
>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>
>>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>
>>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>
>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>> election
>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>> election
>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>> revoked leadership. Clearing fencing
>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>> the
>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>> to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>> to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>> RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>> CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>> RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>> CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>> election
>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-resourcemanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>> to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>> RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>> CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>> RUNNING to
>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>> occurred while renewing lock: Unable to update
>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>> Unable to update ConfigMapLock
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: io.fabric8.kubernetes.client.KubernetesClientException:
>>>>>>> Failure executing: PUT at:
>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>> your changes to the latest version and try again,
>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>> \t... 10 common frames omitted"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>> CANCELING to
>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>> the results produced by task execution
>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>> checkpoint coordinator for job
>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>> 00000000000000000000000000000000 has been
>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>> no longer the
>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>> the JobMaster for job
>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-resourcemanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>> job1-resourcemanager-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>> thread
>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>> granted leadership with fencing token
>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>> the
>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>> collision: Group already contains a Metric with the name
>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>> failed unexpectedly:
>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>
>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>
>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>> error occurred in the cluster
>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>> Application failed unexpectedly.
>>>>>>>
>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>
>>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>
>>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>
>>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>
>>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>> 22 common frames omitted"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>> down rest
>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>> cache directory
>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>> thread
>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>> down
>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>> thread
>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>> the
>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>> the
>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>> closing the watch
>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>> thread
>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>> null
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>
>>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>> watcher is
>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>> websocket
>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>> close received. code: 1000, reason:
>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>> onClose for already closed/closing
>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
Re: Flink application mode using k8S HA.
Posted by David Morávek <dm...@apache.org>.
Hi Marcos, can you please attach full log from the affected JM? The leader
election settings you have seem quite generous.
Best,
D.
On Mon, Dec 13, 2021 at 12:26 PM marco andreas <ma...@gmail.com>
wrote:
> Hello David,
>
> So I am using the official flink setup to deploy a
> standalone application cluster.
>
> For the k8S HA related setting I just changed the lease-period, you will
> find attached the config-map settings.
>
> The reason is that the JM is not able to renew the lease period ,this
> error is totally random , the job may run for hours without any crashes and
> sometimes it starts to crash every 15mn or 30 mn ...
>
> Best,
>
> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a écrit :
>
>> Hi Marco and Mejri,
>>
>> the email didn't get it pass the gmail's spam filter (fairly long
>> json-based log?), which is most likely the reason no-one has picked it up.
>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>
>> What I find most suspicious is this part:
>>
>> Renew deadline reached after 200 seconds while renewing lock
>> ConfigMapLock: job1-flink - job1-restserver-leader
>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>
>> 200 seconds seems like a quite some time for not being able to able to
>> update the config-map with a new lease. Did you change any other k8s /
>> leader election related settings?
>>
>> As for why JM might be gone, in application there is a known bug, that
>> results in the hard-kill when leadership is lost [1].
>>
>> I'm also aware of other issues with the application mode, that has been
>> fixed in 1.13.x and 1.14.x. Also there have been some significant
>> improvements to the Kuberenetes based HA in the latest versions. Is there
>> any chance you can try with the newer version of Flink?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>
>>
>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <me...@gmail.com>
>> wrote:
>>
>>> Any updates on this subject please?
>>>
>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <ma...@gmail.com>
>>> a écrit :
>>>
>>>> Has anyone encountered the same problem and found a solution?
>>>>
>>>> Any suggestions would be appreciated.
>>>>
>>>> Thanks.
>>>>
>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <ma...@gmail.com>
>>>> a écrit :
>>>>
>>>>> I am using the 1.12.5 flink version.
>>>>>
>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <ma...@gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> Hello flink community,
>>>>>>
>>>>>> I am deploying a flink job on a standalone application cluster mode
>>>>>> using kubernetes high availability.
>>>>>>
>>>>>> The job starts running normally but at some point in time it starts
>>>>>> to crash and eventually after 5 restart the jobmanager pod gets deleted.
>>>>>>
>>>>>> Ps: I don't have any memory problems.
>>>>>>
>>>>>> Here's the log of the crashing job.
>>>>>>
>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>> job1-flink - job1-restserver-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>
>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>> http://172.24.10.171:8081 lost
>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>> election
>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>
>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>> job1-flink - job1-dispatcher-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>> dispatcher akka.tcp://flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>> closing the watch
>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>> watcher is
>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>> websocket
>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>> close received. code: 1000, reason:
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>> onClose for already closed/closing
>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>> JobManager is no longer the leader.
>>>>>>
>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>
>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>
>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>
>>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>
>>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>
>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>> election
>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>> election
>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>> revoked leadership. Clearing fencing
>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>> closing the watch
>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>> watcher is
>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>> to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>> websocket
>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>> the
>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>> close received. code: 1000, reason:
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>> onClose for already closed/closing
>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>> to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>> to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>> to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>> to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>> RUNNING to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>> CANCELING to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>> RUNNING to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>> CANCELING to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>> to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>> election
>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>> job1-resourcemanager-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>> to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>> RUNNING to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>> CANCELING to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>> RUNNING to
>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>> occurred while renewing lock: Unable to update
>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>> Unable to update ConfigMapLock
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by: io.fabric8.kubernetes.client.KubernetesClientException:
>>>>>> Failure executing: PUT at:
>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>> your changes to the latest version and try again. Received status:
>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>> your changes to the latest version and try again,
>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>
>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>
>>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>> \t... 10 common frames omitted"}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>> CANCELING to
>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>> the results produced by task execution
>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>> checkpoint coordinator for job
>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>
>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>> 00000000000000000000000000000000 has been
>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>> no longer the
>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>> the JobMaster for job
>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>> job1-resourcemanager-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>> job1-resourcemanager-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>> closing the watch
>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>> watcher is
>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>> thread
>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>> null
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>> websocket
>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>> close received. code: 1000, reason:
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>> onClose for already closed/closing
>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>> granted leadership with fencing token
>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>> the
>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>> collision: Group already contains a Metric with the name
>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>> job graph 00000000000000000000000000000000 from
>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>> failed unexpectedly:
>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>> null
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>
>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>
>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>
>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>
>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>
>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>> error occurred in the cluster
>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>> Application failed unexpectedly.
>>>>>>
>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>
>>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>
>>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>
>>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>
>>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>
>>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>
>>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>> 22 common frames omitted"}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>> down rest
>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>> cache directory
>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>> closing the watch
>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>> watcher is
>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>> thread
>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>> null
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>> websocket
>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>> down
>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>> close received. code: 1000, reason:
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>> onClose for already closed/closing
>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>> closing the watch
>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>> watcher is
>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>> websocket
>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>> close received. code: 1000, reason:
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>> onClose for already closed/closing
>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>> closing the watch
>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>> watcher is
>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>> websocket
>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>> close received. code: 1000, reason:
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>> onClose for already closed/closing
>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>> closing the watch
>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>> watcher is
>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>> thread
>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>> null
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>> websocket
>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>> close received. code: 1000, reason:
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>> onClose for already closed/closing
>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>> the
>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>> the
>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>> closing the watch
>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>> thread
>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>> null
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>
>>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>> watcher is
>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>> websocket
>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>> close received. code: 1000, reason:
>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>> onClose for already closed/closing
>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
Re: Flink application mode using k8S HA.
Posted by marco andreas <ma...@gmail.com>.
Hello David,
So I am using the official flink setup to deploy a
standalone application cluster.
For the k8S HA related setting I just changed the lease-period, you will
find attached the config-map settings.
The reason is that the JM is not able to renew the lease period ,this error
is totally random , the job may run for hours without any crashes and
sometimes it starts to crash every 15mn or 30 mn ...
Best,
Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a écrit :
> Hi Marco and Mejri,
>
> the email didn't get it pass the gmail's spam filter (fairly long
> json-based log?), which is most likely the reason no-one has picked it up.
> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>
> What I find most suspicious is this part:
>
> Renew deadline reached after 200 seconds while renewing lock
> ConfigMapLock: job1-flink - job1-restserver-leader
> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>
> 200 seconds seems like a quite some time for not being able to able to
> update the config-map with a new lease. Did you change any other k8s /
> leader election related settings?
>
> As for why JM might be gone, in application there is a known bug, that
> results in the hard-kill when leadership is lost [1].
>
> I'm also aware of other issues with the application mode, that has been
> fixed in 1.13.x and 1.14.x. Also there have been some significant
> improvements to the Kuberenetes based HA in the latest versions. Is there
> any chance you can try with the newer version of Flink?
>
> [1] https://issues.apache.org/jira/browse/FLINK-23946
>
>
> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <me...@gmail.com>
> wrote:
>
>> Any updates on this subject please?
>>
>> Le mar. 30 nov. 2021 à 20:55, marco andreas <ma...@gmail.com>
>> a écrit :
>>
>>> Has anyone encountered the same problem and found a solution?
>>>
>>> Any suggestions would be appreciated.
>>>
>>> Thanks.
>>>
>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <ma...@gmail.com>
>>> a écrit :
>>>
>>>> I am using the 1.12.5 flink version.
>>>>
>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <ma...@gmail.com>
>>>> a écrit :
>>>>
>>>>> Hello flink community,
>>>>>
>>>>> I am deploying a flink job on a standalone application cluster mode
>>>>> using kubernetes high availability.
>>>>>
>>>>> The job starts running normally but at some point in time it starts to
>>>>> crash and eventually after 5 restart the jobmanager pod gets deleted.
>>>>>
>>>>> Ps: I don't have any memory problems.
>>>>>
>>>>> Here's the log of the crashing job.
>>>>>
>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>> job1-flink - job1-restserver-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>
>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>> http://172.24.10.171:8081 lost
>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>> election
>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>
>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>> job1-flink - job1-dispatcher-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>> dispatcher akka.tcp://flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>> closing the watch
>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>> watcher is
>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>> websocket
>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>> close received. code: 1000, reason:
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>> onClose for already closed/closing
>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>> JobManager is no longer the leader.
>>>>>
>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>
>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>
>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>
>>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>
>>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>
>>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>> election
>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>> election
>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>> job1-flink - job1-resourcemanager-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>> revoked leadership. Clearing fencing
>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>> closing the watch
>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>> watcher is
>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>> to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>> websocket
>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>> the
>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>> close received. code: 1000, reason:
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>> onClose for already closed/closing
>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>> to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>> to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>> to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>> to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>> RUNNING to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>> CANCELING to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>> RUNNING to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>> CANCELING to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>> to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>> election
>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>> job1-resourcemanager-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>> to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>> RUNNING to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>> CANCELING to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>> RUNNING to
>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>> occurred while renewing lock: Unable to update
>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>> Unable to update ConfigMapLock
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: io.fabric8.kubernetes.client.KubernetesClientException:
>>>>> Failure executing: PUT at:
>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>> your changes to the latest version and try again. Received status:
>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>> message=Operation cannot be fulfilled on configmaps
>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>> your changes to the latest version and try again,
>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>
>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>
>>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>
>>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>> \t... 10 common frames omitted"}
>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>> CANCELING to
>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>> the results produced by task execution
>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>> checkpoint coordinator for job
>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>
>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>> 00000000000000000000000000000000 has been
>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>> no longer the
>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>> the JobMaster for job
>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>> job1-resourcemanager-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>> job1-resourcemanager-leader
>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>> closing the watch
>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>> watcher is
>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>> thread
>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>> null
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>> websocket
>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>> close received. code: 1000, reason:
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>> onClose for already closed/closing
>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>> 00000000000000000000000000000000 reached terminal state
>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>> granted leadership with fencing token
>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>> the
>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>> collision: Group already contains a Metric with the name
>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>> Metric will not be reported.[172.24.10.171,
>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>> job graph 00000000000000000000000000000000 from
>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>> failed unexpectedly:
>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>> null
>>>>>
>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>
>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>
>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>
>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>
>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>
>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>> error occurred in the cluster
>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>> Application failed unexpectedly.
>>>>>
>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>
>>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>
>>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>
>>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>
>>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>
>>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>
>>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>
>>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>> 22 common frames omitted"}
>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>> down rest
>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>> cache directory
>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>> closing the watch
>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>> watcher is
>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>> thread
>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>> null
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>> websocket
>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>> down
>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>> close received. code: 1000, reason:
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>> onClose for already closed/closing
>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>> closing the watch
>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>> watcher is
>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>> websocket
>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>> close received. code: 1000, reason:
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>> onClose for already closed/closing
>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>> closing the watch
>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>> watcher is
>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>> websocket
>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>> close received. code: 1000, reason:
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>> onClose for already closed/closing
>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>> closing the watch
>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>> watcher is
>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>> thread
>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>> null
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>> websocket
>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>> close received. code: 1000, reason:
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>> onClose for already closed/closing
>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>> the
>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>> the
>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>> closing the watch
>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>> thread
>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>> null
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>
>>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> java.lang.Thread.run(Thread.java:748)"}
>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>> watcher is
>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>> websocket
>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>> close received. code: 1000, reason:
>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>> onClose for already closed/closing
>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
Re: Flink application mode using k8S HA.
Posted by David Morávek <dm...@apache.org>.
Hi Marco and Mejri,
the email didn't get it pass the gmail's spam filter (fairly long
json-based log?), which is most likely the reason no-one has picked it up.
Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
What I find most suspicious is this part:
Renew deadline reached after 200 seconds while renewing lock ConfigMapLock:
job1-flink - job1-restserver-leader (4176daa0-44c1-4065-9291-5ad09fc6621b)
200 seconds seems like a quite some time for not being able to able to
update the config-map with a new lease. Did you change any other k8s /
leader election related settings?
As for why JM might be gone, in application there is a known bug, that
results in the hard-kill when leadership is lost [1].
I'm also aware of other issues with the application mode, that has been
fixed in 1.13.x and 1.14.x. Also there have been some significant
improvements to the Kuberenetes based HA in the latest versions. Is there
any chance you can try with the newer version of Flink?
[1] https://issues.apache.org/jira/browse/FLINK-23946
On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <me...@gmail.com>
wrote:
> Any updates on this subject please?
>
> Le mar. 30 nov. 2021 à 20:55, marco andreas <ma...@gmail.com> a
> écrit :
>
>> Has anyone encountered the same problem and found a solution?
>>
>> Any suggestions would be appreciated.
>>
>> Thanks.
>>
>> Le lun. 29 nov. 2021 à 18:13, marco andreas <ma...@gmail.com>
>> a écrit :
>>
>>> I am using the 1.12.5 flink version.
>>>
>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <ma...@gmail.com>
>>> a écrit :
>>>
>>>> Hello flink community,
>>>>
>>>> I am deploying a flink job on a standalone application cluster mode
>>>> using kubernetes high availability.
>>>>
>>>> The job starts running normally but at some point in time it starts to
>>>> crash and eventually after 5 restart the jobmanager pod gets deleted.
>>>>
>>>> Ps: I don't have any memory problems.
>>>>
>>>> Here's the log of the crashing job.
>>>>
>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>> job1-flink - job1-restserver-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>> http://172.24.10.171:8081 lost
>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>> election
>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>> http://172.24.10.171:8081 was granted leadership with
>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>> job1-flink - job1-dispatcher-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>> dispatcher akka.tcp://flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>> all currently running jobs of dispatcher akka.tcp://
>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>> closing the watch
>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>> watcher is
>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>> websocket
>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>> close received. code: 1000, reason:
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>> onClose for already closed/closing
>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>> JobManager is no longer the leader.
>>>>
>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>
>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>
>>>> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>
>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>
>>>> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>
>>>> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>
>>>> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>
>>>> org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> java.lang.Thread.run(Thread.java:748)"}
>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>> election
>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>> election
>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>> job1-flink - job1-resourcemanager-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>> revoked leadership. Clearing fencing
>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>> closing the watch
>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>> watcher is
>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>> to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>> websocket
>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>> the
>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>> close received. code: 1000, reason:
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>> onClose for already closed/closing
>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>> to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>> to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>> to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>> to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>> RUNNING to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>> CANCELING to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>> RUNNING to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>> CANCELING to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>> to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>> election
>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>> job1-resourcemanager-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>> to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>> RUNNING to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>> CANCELING to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>> RUNNING to
>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>> occurred while renewing lock: Unable to update
>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>> Unable to update ConfigMapLock
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> java.lang.Thread.run(Thread.java:748)
>>>> Caused by: io.fabric8.kubernetes.client.KubernetesClientException:
>>>> Failure executing: PUT at:
>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>> Message: Operation cannot be fulfilled on configmaps
>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>> your changes to the latest version and try again. Received status:
>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>> message=Operation cannot be fulfilled on configmaps
>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>> your changes to the latest version and try again,
>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>
>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>
>>>> io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>
>>>> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>> \t... 10 common frames omitted"}
>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>> CANCELING to
>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>> the results produced by task execution
>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>> checkpoint coordinator for job
>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>
>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>> 00000000000000000000000000000000 has been
>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>> no longer the
>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>> the JobMaster for job
>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>> job1-resourcemanager-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>> job1-resourcemanager-leader
>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>> closing the watch
>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>> watcher is
>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>> thread
>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>> null
>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> java.lang.Thread.run(Thread.java:748)"}
>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>> websocket
>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>> close received. code: 1000, reason:
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>> onClose for already closed/closing
>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>> 00000000000000000000000000000000 reached terminal state
>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>> granted leadership with fencing token
>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>> the
>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>> collision: Group already contains a Metric with the name
>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>> Metric will not be reported.[172.24.10.171,
>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>> job graph 00000000000000000000000000000000 from
>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>> failed unexpectedly:
>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>> null
>>>>
>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>
>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>
>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>
>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>
>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>
>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>
>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>
>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>
>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>
>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>
>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>
>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> java.lang.Thread.run(Thread.java:748)"}
>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>> error occurred in the cluster
>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>> Application failed unexpectedly.
>>>>
>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>
>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>
>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>
>>>> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>
>>>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>
>>>> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>
>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>
>>>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>
>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>
>>>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>
>>>> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>
>>>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>
>>>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>
>>>> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>
>>>> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>
>>>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> java.lang.Thread.run(Thread.java:748)
>>>> Caused by: java.util.concurrent.CancellationException: null
>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>> 22 common frames omitted"}
>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>> shutdown hook","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>> shutdown hook","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>> down rest
>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>> shutdown hook","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>> cache directory
>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>> closing the watch
>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>> watcher is
>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>> thread
>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>> null
>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> java.lang.Thread.run(Thread.java:748)"}
>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>> websocket
>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>> down
>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>> close received. code: 1000, reason:
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>> onClose for already closed/closing
>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>> closing the watch
>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>> watcher is
>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>> websocket
>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>> close received. code: 1000, reason:
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>> onClose for already closed/closing
>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>> closing the watch
>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>> watcher is
>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>> websocket
>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>> close received. code: 1000, reason:
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>> onClose for already closed/closing
>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>> closing the watch
>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>> watcher is
>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>> thread
>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>> null
>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> java.lang.Thread.run(Thread.java:748)"}
>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>> websocket
>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>> close received. code: 1000, reason:
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>> onClose for already closed/closing
>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>> the
>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>> the
>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>> closing the watch
>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>> thread
>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>> null
>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>
>>>> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> java.lang.Thread.run(Thread.java:748)"}
>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>> watcher is
>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>> websocket
>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>> close received. code: 1000, reason:
>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>> onClose for already closed/closing
>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>
>>>>
>>>>
>>>>
>>>>