You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by marco andreas <ma...@gmail.com> on 2022/01/08 20:32:11 UTC

Re: Flink application mode using k8S HA.

Hello,

I upgraded the flink version lately to 1.13.5, however i encountered the
same problem.

Any thoughts on this issue ?

JM Log in the attached file.

Best regards,


Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com> a
écrit :

> I am sure that the kubernetes.cluster-id is not being reused (it's unique
> ), and I have only one instance of the jobmanager in my cluster.
>
> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a écrit :
>
>> Is it possible that you might be reusing the same kubernetes.cluster-id
>> (job1) between multiple flink clusters? How many jobmanagers are you
>> running for the cluster?
>>
>> D.
>>
>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <ma...@gmail.com>
>> wrote:
>>
>>> Hi David,
>>>
>>> I dont have the full log.
>>> But here's the full log of a crashed JM with similar configurations
>>> (just the lease period is different : 60 s )
>>>
>>> Best,
>>>
>>>
>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a écrit :
>>>
>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>> leader election settings you have seem quite generous.
>>>>
>>>> Best,
>>>> D.
>>>>
>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>> marcoandreas582@gmail.com> wrote:
>>>>
>>>>> Hello David,
>>>>>
>>>>> So I am using the official flink setup to deploy a
>>>>> standalone application cluster.
>>>>>
>>>>> For the k8S HA related setting I just changed the lease-period, you
>>>>> will find attached the config-map settings.
>>>>>
>>>>> The reason is that the JM is not able to renew the lease period ,this
>>>>> error is totally random , the job may run for hours without any crashes and
>>>>> sometimes it starts to crash every 15mn or 30 mn ...
>>>>>
>>>>> Best,
>>>>>
>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>> écrit :
>>>>>
>>>>>> Hi Marco and Mejri,
>>>>>>
>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>
>>>>>> What I find most suspicious is this part:
>>>>>>
>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>
>>>>>> 200 seconds seems like a quite some time for not being able to able
>>>>>> to update the config-map with a new lease. Did you change any other k8s /
>>>>>> leader election related settings?
>>>>>>
>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>
>>>>>> I'm also aware of other issues with the application mode, that has
>>>>>> been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>> any chance you can try with the newer version of Flink?
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>
>>>>>>> Any updates on this subject please?
>>>>>>>
>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>
>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>
>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>
>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>
>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>>> Hello flink community,
>>>>>>>>>>
>>>>>>>>>> I am  deploying a flink job on a standalone application cluster
>>>>>>>>>> mode using kubernetes high availability.
>>>>>>>>>>
>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>> deleted.
>>>>>>>>>>
>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>
>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>
>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>> election
>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>> election
>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>> election
>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>> was revoked leadership. Clearing fencing
>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>> the
>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>> to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>> to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>> RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>> CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>> RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>> CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>> election
>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>> to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>> RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>> CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>> RUNNING to
>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>  java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>> Caused by:
>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>> PUT at:
>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>> CANCELING to
>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>> the results produced by task execution
>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>> no longer the
>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>> the JobMaster for job
>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>> thread
>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>> was granted leadership with fencing token
>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>> the
>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>> failed unexpectedly:
>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>> error occurred in the cluster
>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>
>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>> down rest
>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>> cache directory
>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>> thread
>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>> down
>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>> thread
>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>> the
>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>> the
>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>> closing the watch
>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>> thread
>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>> null
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>
>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>
>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>> watcher is
>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>> websocket
>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>

Re: Flink application mode using k8S HA.

Posted by Yang Wang <da...@gmail.com>.
>
> What I don't understand is why the application gets shut down when the api
> server is not responding.


The root cause is the Flink JobManager/ResourceManager lost the leadership
and then will restart the JobManager process.

If you could reproduce the issue in a minikube environment, then it will be
easier to debug.
Refer to [1] for how to enable the APIServer audit log for minikube.

[1]. https://minikube.sigs.k8s.io/docs/tutorials/audit-policy/

Best,
Yang

marco andreas <ma...@gmail.com> 于2022年1月28日周五 01:20写道:

> Hello ,
>
> With respect to the memory issue , I have no problem.
>
> Regarding the k8S APISERVER, I can't check the log because it's
> a mutualised cluster and I don't have the rights to do so .
>
> What I don't understand is why the application gets shut down when the api
> server is not responding.
>
> In the meantime do you have any solutions to get around this problem by
> the time I can get the api server log ?
>
> Thanks,
>
>
>
> Le mar. 11 janv. 2022 à 04:11, Yang Wang <da...@gmail.com> a écrit :
>
>> You could add the following config option to enable GC log.
>>
>> env.java.opts.jobmanager: -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/opt/flink/log/jobmanager-gc.log
>>
>> Given that your JobManager heap is not too large(> 20Gb), the 300s fullGC
>> should not happen. But we could enable the GC log for double checking.
>>
>>
>> Another suspicion is that K8s APIServer might be overloaded. You need to
>> check the K8s APIServer audit log to confirm whether the renew ConfigMap
>> request has been received.
>>
>>
>> Best,
>> Yang
>>
>>
>> marco andreas <ma...@gmail.com> 于2022年1月11日周二 02:22写道:
>>
>>> Hello Yang,
>>>
>>> Yes, that's the problem. The jobmanager wast'n able to renew the lease
>>> period despite the 300s of time I had configured.
>>>
>>> Could you please tell how to verify how I can tell that the JobManager
>>> pod was having a long fullGC.
>>>
>>> Best,
>>>
>>> Le lun. 10 janv. 2022 à 04:42, Yang Wang <da...@gmail.com> a
>>> écrit :
>>>
>>>> The job failed because of "Renew deadline reached after 300 seconds
>>>> while renewing lock".
>>>>
>>>> Could you please verify that the JobManager pod was having a long
>>>> fullGC or something is wrong with the network from 2022-01-08T15:34:22 to
>>>> 2022-01-08T15:37:07?
>>>>
>>>>
>>>>
>>>>
>>>> Best,
>>>> Yang
>>>>
>>>> marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道:
>>>>
>>>>> Hello,
>>>>>
>>>>> I upgraded the flink version lately to 1.13.5, however i
>>>>> encountered the same problem.
>>>>>
>>>>> Any thoughts on this issue ?
>>>>>
>>>>> JM Log in the attached file.
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> I am sure that the kubernetes.cluster-id is not being reused (it's
>>>>>> unique ), and I have only one instance of the jobmanager in my cluster.
>>>>>>
>>>>>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a
>>>>>> écrit :
>>>>>>
>>>>>>> Is it possible that you might be reusing the same
>>>>>>> kubernetes.cluster-id (job1) between multiple flink clusters? How many
>>>>>>> jobmanagers are you running for the cluster?
>>>>>>>
>>>>>>> D.
>>>>>>>
>>>>>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <
>>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi David,
>>>>>>>>
>>>>>>>> I dont have the full log.
>>>>>>>> But here's the full log of a crashed JM with similar configurations
>>>>>>>> (just the lease period is different : 60 s )
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>>
>>>>>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> Hi Marcos, can you please attach full log from the affected JM?
>>>>>>>>> The leader election settings you have seem quite generous.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello David,
>>>>>>>>>>
>>>>>>>>>> So I am using the official flink setup to deploy a
>>>>>>>>>> standalone application cluster.
>>>>>>>>>>
>>>>>>>>>> For the k8S HA related setting I just changed the lease-period,
>>>>>>>>>> you will find attached the config-map settings.
>>>>>>>>>>
>>>>>>>>>> The reason is that the JM is not able to renew the lease
>>>>>>>>>> period ,this error is totally random , the job may run for hours without
>>>>>>>>>> any crashes and sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>>>>>> écrit :
>>>>>>>>>>
>>>>>>>>>>> Hi Marco and Mejri,
>>>>>>>>>>>
>>>>>>>>>>> the email didn't get it pass the gmail's spam filter (fairly
>>>>>>>>>>> long json-based log?), which is most likely the reason no-one has picked it
>>>>>>>>>>> up. Can you please elaborate bit more about your setup (scale, k8s cluster,
>>>>>>>>>>> ...)
>>>>>>>>>>>
>>>>>>>>>>> What I find most suspicious is this part:
>>>>>>>>>>>
>>>>>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>>>>>
>>>>>>>>>>> 200 seconds seems like a quite some time for not being able to
>>>>>>>>>>> able to update the config-map with a new lease. Did you change any other
>>>>>>>>>>> k8s / leader election related settings?
>>>>>>>>>>>
>>>>>>>>>>> As for why JM might be gone, in application there is a known
>>>>>>>>>>> bug, that results in the hard-kill when leadership is lost [1].
>>>>>>>>>>>
>>>>>>>>>>> I'm also aware of other issues with the application mode, that
>>>>>>>>>>> has been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>>>>>
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Any updates on this subject please?
>>>>>>>>>>>>
>>>>>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am  deploying a flink job on a standalone application
>>>>>>>>>>>>>>> cluster mode using kubernetes high availability.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>>>>>> deleted.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>>> election
>>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>>> election
>>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>>> election
>>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>>>> akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>>>>>>>>> revoked leadership. Clearing fencing
>>>>>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>>> election
>>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>>>>>  java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>>>>>> PUT at:
>>>>>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>>>>>> no longer the
>>>>>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>>>> akka.tcp://
>>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>>>>>>>>> granted leadership with fencing token
>>>>>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>>> down rest
>>>>>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>>>>>> cache directory
>>>>>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>>>>>> down
>>>>>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>>> https://172.31.64.1/..
>>>>>>>>>>>>>>> .","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>

Re: Flink application mode using k8S HA.

Posted by marco andreas <ma...@gmail.com>.
Hello ,

With respect to the memory issue , I have no problem.

Regarding the k8S APISERVER, I can't check the log because it's
a mutualised cluster and I don't have the rights to do so .

What I don't understand is why the application gets shut down when the api
server is not responding.

In the meantime do you have any solutions to get around this problem by
the time I can get the api server log ?

Thanks,



Le mar. 11 janv. 2022 à 04:11, Yang Wang <da...@gmail.com> a écrit :

> You could add the following config option to enable GC log.
>
> env.java.opts.jobmanager: -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/opt/flink/log/jobmanager-gc.log
>
> Given that your JobManager heap is not too large(> 20Gb), the 300s fullGC
> should not happen. But we could enable the GC log for double checking.
>
>
> Another suspicion is that K8s APIServer might be overloaded. You need to
> check the K8s APIServer audit log to confirm whether the renew ConfigMap
> request has been received.
>
>
> Best,
> Yang
>
>
> marco andreas <ma...@gmail.com> 于2022年1月11日周二 02:22写道:
>
>> Hello Yang,
>>
>> Yes, that's the problem. The jobmanager wast'n able to renew the lease
>> period despite the 300s of time I had configured.
>>
>> Could you please tell how to verify how I can tell that the JobManager
>> pod was having a long fullGC.
>>
>> Best,
>>
>> Le lun. 10 janv. 2022 à 04:42, Yang Wang <da...@gmail.com> a
>> écrit :
>>
>>> The job failed because of "Renew deadline reached after 300 seconds
>>> while renewing lock".
>>>
>>> Could you please verify that the JobManager pod was having a long fullGC
>>> or something is wrong with the network from 2022-01-08T15:34:22 to
>>> 2022-01-08T15:37:07?
>>>
>>>
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道:
>>>
>>>> Hello,
>>>>
>>>> I upgraded the flink version lately to 1.13.5, however i
>>>> encountered the same problem.
>>>>
>>>> Any thoughts on this issue ?
>>>>
>>>> JM Log in the attached file.
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com>
>>>> a écrit :
>>>>
>>>>> I am sure that the kubernetes.cluster-id is not being reused (it's
>>>>> unique ), and I have only one instance of the jobmanager in my cluster.
>>>>>
>>>>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a
>>>>> écrit :
>>>>>
>>>>>> Is it possible that you might be reusing the same
>>>>>> kubernetes.cluster-id (job1) between multiple flink clusters? How many
>>>>>> jobmanagers are you running for the cluster?
>>>>>>
>>>>>> D.
>>>>>>
>>>>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <
>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>
>>>>>>> Hi David,
>>>>>>>
>>>>>>> I dont have the full log.
>>>>>>> But here's the full log of a crashed JM with similar configurations
>>>>>>> (just the lease period is different : 60 s )
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>>
>>>>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>>>>>> leader election settings you have seem quite generous.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> D.
>>>>>>>>
>>>>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello David,
>>>>>>>>>
>>>>>>>>> So I am using the official flink setup to deploy a
>>>>>>>>> standalone application cluster.
>>>>>>>>>
>>>>>>>>> For the k8S HA related setting I just changed the lease-period,
>>>>>>>>> you will find attached the config-map settings.
>>>>>>>>>
>>>>>>>>> The reason is that the JM is not able to renew the lease
>>>>>>>>> period ,this error is totally random , the job may run for hours without
>>>>>>>>> any crashes and sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>>> Hi Marco and Mejri,
>>>>>>>>>>
>>>>>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>>>>>
>>>>>>>>>> What I find most suspicious is this part:
>>>>>>>>>>
>>>>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>>>>
>>>>>>>>>> 200 seconds seems like a quite some time for not being able to
>>>>>>>>>> able to update the config-map with a new lease. Did you change any other
>>>>>>>>>> k8s / leader election related settings?
>>>>>>>>>>
>>>>>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>>>>>
>>>>>>>>>> I'm also aware of other issues with the application mode, that
>>>>>>>>>> has been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Any updates on this subject please?
>>>>>>>>>>>
>>>>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>>>>
>>>>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am  deploying a flink job on a standalone application
>>>>>>>>>>>>>> cluster mode using kubernetes high availability.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>>>>> deleted.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>> election
>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>> election
>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>> election
>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>>> akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>>>>>>>> revoked leadership. Clearing fencing
>>>>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>>> election
>>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>>>>  java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>>>>> PUT at:
>>>>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>>>>> no longer the
>>>>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>>> akka.tcp://
>>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/resourcemanager_0 was
>>>>>>>>>>>>>> granted leadership with fencing token
>>>>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>>> down rest
>>>>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>>>>> cache directory
>>>>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>>>>> down
>>>>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>>> websocket
>>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>

Re: Flink application mode using k8S HA.

Posted by Yang Wang <da...@gmail.com>.
You could add the following config option to enable GC log.

env.java.opts.jobmanager: -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/opt/flink/log/jobmanager-gc.log

Given that your JobManager heap is not too large(> 20Gb), the 300s fullGC
should not happen. But we could enable the GC log for double checking.


Another suspicion is that K8s APIServer might be overloaded. You need to
check the K8s APIServer audit log to confirm whether the renew ConfigMap
request has been received.


Best,
Yang


marco andreas <ma...@gmail.com> 于2022年1月11日周二 02:22写道:

> Hello Yang,
>
> Yes, that's the problem. The jobmanager wast'n able to renew the lease
> period despite the 300s of time I had configured.
>
> Could you please tell how to verify how I can tell that the JobManager pod
> was having a long fullGC.
>
> Best,
>
> Le lun. 10 janv. 2022 à 04:42, Yang Wang <da...@gmail.com> a écrit :
>
>> The job failed because of "Renew deadline reached after 300 seconds while
>> renewing lock".
>>
>> Could you please verify that the JobManager pod was having a long fullGC
>> or something is wrong with the network from 2022-01-08T15:34:22 to
>> 2022-01-08T15:37:07?
>>
>>
>>
>>
>> Best,
>> Yang
>>
>> marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道:
>>
>>> Hello,
>>>
>>> I upgraded the flink version lately to 1.13.5, however i encountered the
>>> same problem.
>>>
>>> Any thoughts on this issue ?
>>>
>>> JM Log in the attached file.
>>>
>>> Best regards,
>>>
>>>
>>> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com>
>>> a écrit :
>>>
>>>> I am sure that the kubernetes.cluster-id is not being reused (it's
>>>> unique ), and I have only one instance of the jobmanager in my cluster.
>>>>
>>>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a écrit :
>>>>
>>>>> Is it possible that you might be reusing the same
>>>>> kubernetes.cluster-id (job1) between multiple flink clusters? How many
>>>>> jobmanagers are you running for the cluster?
>>>>>
>>>>> D.
>>>>>
>>>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <
>>>>> marcoandreas582@gmail.com> wrote:
>>>>>
>>>>>> Hi David,
>>>>>>
>>>>>> I dont have the full log.
>>>>>> But here's the full log of a crashed JM with similar configurations
>>>>>> (just the lease period is different : 60 s )
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>>
>>>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a
>>>>>> écrit :
>>>>>>
>>>>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>>>>> leader election settings you have seem quite generous.
>>>>>>>
>>>>>>> Best,
>>>>>>> D.
>>>>>>>
>>>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello David,
>>>>>>>>
>>>>>>>> So I am using the official flink setup to deploy a
>>>>>>>> standalone application cluster.
>>>>>>>>
>>>>>>>> For the k8S HA related setting I just changed the lease-period, you
>>>>>>>> will find attached the config-map settings.
>>>>>>>>
>>>>>>>> The reason is that the JM is not able to renew the lease
>>>>>>>> period ,this error is totally random , the job may run for hours without
>>>>>>>> any crashes and sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> Hi Marco and Mejri,
>>>>>>>>>
>>>>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>>>>
>>>>>>>>> What I find most suspicious is this part:
>>>>>>>>>
>>>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>>>
>>>>>>>>> 200 seconds seems like a quite some time for not being able to
>>>>>>>>> able to update the config-map with a new lease. Did you change any other
>>>>>>>>> k8s / leader election related settings?
>>>>>>>>>
>>>>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>>>>
>>>>>>>>> I'm also aware of other issues with the application mode, that has
>>>>>>>>> been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>>>
>>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Any updates on this subject please?
>>>>>>>>>>
>>>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>>>
>>>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>>>
>>>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am  deploying a flink job on a standalone application
>>>>>>>>>>>>> cluster mode using kubernetes high availability.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>>>> deleted.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>>>
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>> election
>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>> election
>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>> election
>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>>>> was revoked leadership. Clearing fencing
>>>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>>>> election
>>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>>>> to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>>>  java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>>>> PUT at:
>>>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>>
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>>>> no longer the
>>>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>> thread
>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>>>> was granted leadership with fencing token
>>>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>>> down rest
>>>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>>>> cache directory
>>>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>> thread
>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>>>> down
>>>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>> thread
>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>>>> closing the watch
>>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>>>> thread
>>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>>
>>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>>>> watcher is
>>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>>> websocket
>>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>

Re: Flink application mode using k8S HA.

Posted by marco andreas <ma...@gmail.com>.
Hello Yang,

Yes, that's the problem. The jobmanager wast'n able to renew the lease
period despite the 300s of time I had configured.

Could you please tell how to verify how I can tell that the JobManager pod
was having a long fullGC.

Best,

Le lun. 10 janv. 2022 à 04:42, Yang Wang <da...@gmail.com> a écrit :

> The job failed because of "Renew deadline reached after 300 seconds while
> renewing lock".
>
> Could you please verify that the JobManager pod was having a long fullGC
> or something is wrong with the network from 2022-01-08T15:34:22 to
> 2022-01-08T15:37:07?
>
>
>
>
> Best,
> Yang
>
> marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道:
>
>> Hello,
>>
>> I upgraded the flink version lately to 1.13.5, however i encountered the
>> same problem.
>>
>> Any thoughts on this issue ?
>>
>> JM Log in the attached file.
>>
>> Best regards,
>>
>>
>> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com>
>> a écrit :
>>
>>> I am sure that the kubernetes.cluster-id is not being reused (it's
>>> unique ), and I have only one instance of the jobmanager in my cluster.
>>>
>>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a écrit :
>>>
>>>> Is it possible that you might be reusing the same kubernetes.cluster-id
>>>> (job1) between multiple flink clusters? How many jobmanagers are you
>>>> running for the cluster?
>>>>
>>>> D.
>>>>
>>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <
>>>> marcoandreas582@gmail.com> wrote:
>>>>
>>>>> Hi David,
>>>>>
>>>>> I dont have the full log.
>>>>> But here's the full log of a crashed JM with similar configurations
>>>>> (just the lease period is different : 60 s )
>>>>>
>>>>> Best,
>>>>>
>>>>>
>>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a
>>>>> écrit :
>>>>>
>>>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>>>> leader election settings you have seem quite generous.
>>>>>>
>>>>>> Best,
>>>>>> D.
>>>>>>
>>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>>> marcoandreas582@gmail.com> wrote:
>>>>>>
>>>>>>> Hello David,
>>>>>>>
>>>>>>> So I am using the official flink setup to deploy a
>>>>>>> standalone application cluster.
>>>>>>>
>>>>>>> For the k8S HA related setting I just changed the lease-period, you
>>>>>>> will find attached the config-map settings.
>>>>>>>
>>>>>>> The reason is that the JM is not able to renew the lease
>>>>>>> period ,this error is totally random , the job may run for hours without
>>>>>>> any crashes and sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Hi Marco and Mejri,
>>>>>>>>
>>>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>>>
>>>>>>>> What I find most suspicious is this part:
>>>>>>>>
>>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>>
>>>>>>>> 200 seconds seems like a quite some time for not being able to able
>>>>>>>> to update the config-map with a new lease. Did you change any other k8s /
>>>>>>>> leader election related settings?
>>>>>>>>
>>>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>>>
>>>>>>>> I'm also aware of other issues with the application mode, that has
>>>>>>>> been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Any updates on this subject please?
>>>>>>>>>
>>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>>
>>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>>
>>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>>
>>>>>>>>>>>> I am  deploying a flink job on a standalone application cluster
>>>>>>>>>>>> mode using kubernetes high availability.
>>>>>>>>>>>>
>>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>>> deleted.
>>>>>>>>>>>>
>>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>>
>>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>>
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>>> election
>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>>> election
>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>>> election
>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>>> was revoked leadership. Clearing fencing
>>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>> the
>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>>> election
>>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>>> to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>> RUNNING to
>>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>>  java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>> Caused by:
>>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>>> PUT at:
>>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>>> CANCELING to
>>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>>
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>>> no longer the
>>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>>> thread
>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>>> was granted leadership with fencing token
>>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>>> the
>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>>
>>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>>> down rest
>>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>>> cache directory
>>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>>> thread
>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>>> down
>>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>>> thread
>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> the
>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>>> the
>>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>>> closing the watch
>>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>>> thread
>>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>>
>>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>>
>>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>>> watcher is
>>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>>> websocket
>>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>

Re: Flink application mode using k8S HA.

Posted by Yang Wang <da...@gmail.com>.
The job failed because of "Renew deadline reached after 300 seconds while
renewing lock".

Could you please verify that the JobManager pod was having a long fullGC or
something is wrong with the network from 2022-01-08T15:34:22 to
2022-01-08T15:37:07?




Best,
Yang

marco andreas <ma...@gmail.com> 于2022年1月9日周日 04:32写道:

> Hello,
>
> I upgraded the flink version lately to 1.13.5, however i encountered the
> same problem.
>
> Any thoughts on this issue ?
>
> JM Log in the attached file.
>
> Best regards,
>
>
> Le lun. 13 déc. 2021 à 17:05, marco andreas <ma...@gmail.com> a
> écrit :
>
>> I am sure that the kubernetes.cluster-id is not being reused (it's unique
>> ), and I have only one instance of the jobmanager in my cluster.
>>
>> Le lun. 13 déc. 2021 à 15:14, David Morávek <dm...@apache.org> a écrit :
>>
>>> Is it possible that you might be reusing the same kubernetes.cluster-id
>>> (job1) between multiple flink clusters? How many jobmanagers are you
>>> running for the cluster?
>>>
>>> D.
>>>
>>> On Mon, Dec 13, 2021 at 2:37 PM marco andreas <ma...@gmail.com>
>>> wrote:
>>>
>>>> Hi David,
>>>>
>>>> I dont have the full log.
>>>> But here's the full log of a crashed JM with similar configurations
>>>> (just the lease period is different : 60 s )
>>>>
>>>> Best,
>>>>
>>>>
>>>> Le lun. 13 déc. 2021 à 13:28, David Morávek <dm...@apache.org> a écrit :
>>>>
>>>>> Hi Marcos, can you please attach full log from the affected JM? The
>>>>> leader election settings you have seem quite generous.
>>>>>
>>>>> Best,
>>>>> D.
>>>>>
>>>>> On Mon, Dec 13, 2021 at 12:26 PM marco andreas <
>>>>> marcoandreas582@gmail.com> wrote:
>>>>>
>>>>>> Hello David,
>>>>>>
>>>>>> So I am using the official flink setup to deploy a
>>>>>> standalone application cluster.
>>>>>>
>>>>>> For the k8S HA related setting I just changed the lease-period, you
>>>>>> will find attached the config-map settings.
>>>>>>
>>>>>> The reason is that the JM is not able to renew the lease period ,this
>>>>>> error is totally random , the job may run for hours without any crashes and
>>>>>> sometimes it starts to crash every 15mn or 30 mn ...
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Le lun. 13 déc. 2021 à 09:52, David Morávek <dm...@apache.org> a
>>>>>> écrit :
>>>>>>
>>>>>>> Hi Marco and Mejri,
>>>>>>>
>>>>>>> the email didn't get it pass the gmail's spam filter (fairly long
>>>>>>> json-based log?), which is most likely the reason no-one has picked it up.
>>>>>>> Can you please elaborate bit more about your setup (scale, k8s cluster, ...)
>>>>>>>
>>>>>>> What I find most suspicious is this part:
>>>>>>>
>>>>>>> Renew deadline reached after 200 seconds while renewing lock
>>>>>>> ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)
>>>>>>>
>>>>>>> 200 seconds seems like a quite some time for not being able to able
>>>>>>> to update the config-map with a new lease. Did you change any other k8s /
>>>>>>> leader election related settings?
>>>>>>>
>>>>>>> As for why JM might be gone, in application there is a known bug,
>>>>>>> that results in the hard-kill when leadership is lost [1].
>>>>>>>
>>>>>>> I'm also aware of other issues with the application mode, that has
>>>>>>> been fixed in 1.13.x and 1.14.x. Also there have been some significant
>>>>>>> improvements to the Kuberenetes based HA in the latest versions. Is there
>>>>>>> any chance you can try with the newer version of Flink?
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-23946
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Dec 12, 2021 at 12:25 PM mejri houssem <
>>>>>>> mejrihoussem09@gmail.com> wrote:
>>>>>>>
>>>>>>>> Any updates on this subject please?
>>>>>>>>
>>>>>>>> Le mar. 30 nov. 2021 à 20:55, marco andreas <
>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>
>>>>>>>>> Has anyone encountered the same problem and found a solution?
>>>>>>>>>
>>>>>>>>> Any suggestions would be appreciated.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Le lun. 29 nov. 2021 à 18:13, marco andreas <
>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>>> I am using the 1.12.5 flink version.
>>>>>>>>>>
>>>>>>>>>> Le lun. 29 nov. 2021 à 10:41, marco andreas <
>>>>>>>>>> marcoandreas582@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>>> Hello flink community,
>>>>>>>>>>>
>>>>>>>>>>> I am  deploying a flink job on a standalone application cluster
>>>>>>>>>>> mode using kubernetes high availability.
>>>>>>>>>>>
>>>>>>>>>>> The job starts running normally but at some point in time it
>>>>>>>>>>> starts to crash and eventually after 5 restart the jobmanager pod gets
>>>>>>>>>>> deleted.
>>>>>>>>>>>
>>>>>>>>>>> Ps: I don't have any memory problems.
>>>>>>>>>>>
>>>>>>>>>>> Here's the log of the crashing job.
>>>>>>>>>>>
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:01:08.137+01:00","@version":"1","message":"Completed
>>>>>>>>>>> checkpoint 201 for job 00000000000000000000000000000000 (474 bytes in 147
>>>>>>>>>>> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.799+01:00","@version":"1","message":"Renew
>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>> job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-7159-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.800+01:00","@version":"1","message":"
>>>>>>>>>>> http://172.24.10.171:8081 lost
>>>>>>>>>>> leadership","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Leader
>>>>>>>>>>> election
>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.824+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"Successfully
>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-47997-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>>
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.837+01:00","@version":"1","message":"
>>>>>>>>>>> http://172.24.10.171:8081 was granted leadership with
>>>>>>>>>>> leaderSessionID=2cd37690-662e-42e0-aecd-2c6ec82b3354","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:21.850+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-restserver-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>> job1-flink - job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-16-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.418+01:00","@version":"1","message":"Renew
>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>> job1-flink - job1-dispatcher-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"JobManager
>>>>>>>>>>> for job job1 (00000000000000000000000000000000) at akka.tcp://
>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/jobmanager_2 was revoked
>>>>>>>>>>> leadership.","logger_name":"org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.420+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.421+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> dispatcher akka.tcp://
>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> all currently running jobs of dispatcher akka.tcp://
>>>>>>>>>>> flink@172.24.10.171:6123/user/rpc/dispatcher_1
>>>>>>>>>>> .","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.422+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@50f5ce90
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.423+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@20fee8d1
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.428+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.429+01:00","@version":"1","message":"Job
>>>>>>>>>>> job1 (00000000000000000000000000000000) switched from state RUNNING to
>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>> JobManager is no longer the leader.
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeJobMasterLeadership(JobManagerRunnerImpl.java:453)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$null$5(JobManagerRunnerImpl.java:435)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$revokeLeadership$6(JobManagerRunnerImpl.java:432)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.revokeLeadership(JobManagerRunnerImpl.java:419)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadership(DefaultLeaderElectionService.java:239)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:213)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Leader
>>>>>>>>>>> election
>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.437+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.439+01:00","@version":"1","message":"Leader
>>>>>>>>>>> election
>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.515+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.519+01:00","@version":"1","message":"Renew
>>>>>>>>>>> deadline reached after 200 seconds while renewing lock ConfigMapLock:
>>>>>>>>>>> job1-flink - job1-resourcemanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.521+01:00","@version":"1","message":"Source:
>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>> was revoked leadership. Clearing fencing
>>>>>>>>>>> token.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@29dfe5bc
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.522+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Source:
>>>>>>>>>>> Rabbit MQ (1/1) (d6df8cc035cfbb727790be7f10247743) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a38b127
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-24","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"Suspending
>>>>>>>>>>> the
>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.523+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.524+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.526+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> d6df8cc035cfbb727790be7f10247743.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.528+01:00","@version":"1","message":"Successfully
>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48000-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink - job1-dispatcher-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.529+01:00","@version":"1","message":"Yaml
>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from RUNNING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Yaml
>>>>>>>>>>> concersion (1/2) (f8e0a9e7c6129604a8f803749861c02a) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Successfully
>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48001-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.615+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-00000000000000000000000000000000-jobmanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> f8e0a9e7c6129604a8f803749861c02a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from RUNNING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.616+01:00","@version":"1","message":"Yaml
>>>>>>>>>>> concersion (2/2) (68885e6d48ff91c1f892095806cfe64d) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.617+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 68885e6d48ff91c1f892095806cfe64d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.618+01:00","@version":"1","message":"Avro
>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>> RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>> schema filtering (1/2) (beb42c8033aef23e77a681d50248038e) switched from
>>>>>>>>>>> CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> beb42c8033aef23e77a681d50248038e.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>> RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.619+01:00","@version":"1","message":"Avro
>>>>>>>>>>> schema filtering (2/2) (7ca0429b0f0cbf782704cecabbc2a972) switched from
>>>>>>>>>>> CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.620+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 7ca0429b0f0cbf782704cecabbc2a972.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.622+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>> states (1/2) (19589e585ec6d14cbe07326ef7f87c77) switched from CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.717+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 19589e585ec6d14cbe07326ef7f87c77.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.718+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Retrieve
>>>>>>>>>>> states (2/2) (f367f6e9d13ad4e261df457b0f706b02) switched from CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.719+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> f367f6e9d13ad4e261df457b0f706b02.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.721+01:00","@version":"1","message":"retrieve
>>>>>>>>>>> All users (1/2) (62d674a839018fd6e77e4677827b0469) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.722+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 62d674a839018fd6e77e4677827b0469.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Leader
>>>>>>>>>>> election
>>>>>>>>>>> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to acquire leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"retrieve
>>>>>>>>>>> All users (2/2) (870c3eda35ccedd92f6e14043c28a8c5) switched from CANCELING
>>>>>>>>>>> to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.723+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 870c3eda35ccedd92f6e14043c28a8c5.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>> RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.724+01:00","@version":"1","message":"Sink:
>>>>>>>>>>> Send to sender (1/2) (6b4d43e1bc854d1c55c4daa41b2e695f) switched from
>>>>>>>>>>> CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 6b4d43e1bc854d1c55c4daa41b2e695f.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Sink:
>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>> RUNNING to
>>>>>>>>>>> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.726+01:00","@version":"1","message":"Exception
>>>>>>>>>>> occurred while renewing lock: Unable to update
>>>>>>>>>>> ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48002-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
>>>>>>>>>>> Unable to update ConfigMapLock
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>  java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>> Caused by:
>>>>>>>>>>> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
>>>>>>>>>>> PUT at:
>>>>>>>>>>> https://172.31.64.1/api/v1/namespaces/job1-flink/configmaps/job1-resourcemanager-leader.
>>>>>>>>>>> Message: Operation cannot be fulfilled on configmaps
>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>> your changes to the latest version and try again. Received status:
>>>>>>>>>>> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[],
>>>>>>>>>>> group=null, kind=configmaps, name=job1-resourcemanager-leader,
>>>>>>>>>>> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
>>>>>>>>>>> message=Operation cannot be fulfilled on configmaps
>>>>>>>>>>> \"job1-resourcemanager-leader\": the object has been modified; please apply
>>>>>>>>>>> your changes to the latest version and try again,
>>>>>>>>>>> metadata=ListMeta(_continue=null, remainingItemCount=null,
>>>>>>>>>>> resourceVersion=null, selfLink=null, additionalProperties={}),
>>>>>>>>>>> reason=Conflict, status=Failure, additionalProperties={}).
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)
>>>>>>>>>>> \t... 10 common frames omitted"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Sink:
>>>>>>>>>>> Send to sender (2/2) (0d7d704c247b75a0e9bb95fa7bc7db61) switched from
>>>>>>>>>>> CANCELING to
>>>>>>>>>>> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.727+01:00","@version":"1","message":"Discarding
>>>>>>>>>>> the results produced by task execution
>>>>>>>>>>> 0d7d704c247b75a0e9bb95fa7bc7db61.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.728+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> checkpoint coordinator for job
>>>>>>>>>>> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>>
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.729+01:00","@version":"1","message":"Shutting
>>>>>>>>>>> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.730+01:00","@version":"1","message":"Job
>>>>>>>>>>> 00000000000000000000000000000000 has been
>>>>>>>>>>> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.734+01:00","@version":"1","message":"Suspending
>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Close
>>>>>>>>>>> ResourceManager connection f0c119f45370531abe540ea7a02408b1: JobManager is
>>>>>>>>>>> no longer the
>>>>>>>>>>> leader..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> the JobMaster for job
>>>>>>>>>>> job1(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.735+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.738+01:00","@version":"1","message":"Successfully
>>>>>>>>>>> Acquired leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-48007-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.816+01:00","@version":"1","message":"Attempting
>>>>>>>>>>> to renew leader lease 'ConfigMapLock: job1-flink -
>>>>>>>>>>> job1-resourcemanager-leader
>>>>>>>>>>> (4176daa0-44c1-4065-9291-5ad09fc6621b)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Closing
>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1c3a2f02
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.817+01:00","@version":"1","message":"Loop
>>>>>>>>>>> thread
>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.818+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1c0555e1
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.819+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Job
>>>>>>>>>>> 00000000000000000000000000000000 reached terminal state
>>>>>>>>>>> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-24","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"ResourceManager
>>>>>>>>>>> akka.tcp://flink@172.24.10.171:6123/user/rpc/resourcemanager_0
>>>>>>>>>>> was granted leadership with fencing token
>>>>>>>>>>> 867de6dd2527b265f63c42f07e0d4ed2","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.820+01:00","@version":"1","message":"Starting
>>>>>>>>>>> the
>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-16","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>> collision: Group already contains a Metric with the name
>>>>>>>>>>> 'taskSlotsAvailable'. Metric will not be reported.[172.24.10.171,
>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Name
>>>>>>>>>>> collision: Group already contains a Metric with the name 'taskSlotsTotal'.
>>>>>>>>>>> Metric will not be reported.[172.24.10.171,
>>>>>>>>>>> jobmanager]","logger_name":"org.apache.flink.metrics.MetricGroup","thread_name":"flink-akka.actor.default-dispatcher-16","level":"WARN","level_value":30000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.821+01:00","@version":"1","message":"Released
>>>>>>>>>>> job graph 00000000000000000000000000000000 from
>>>>>>>>>>> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-1","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.822+01:00","@version":"1","message":"Application
>>>>>>>>>>> failed unexpectedly:
>>>>>>>>>>> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-1","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.824+01:00","@version":"1","message":"Fatal
>>>>>>>>>>> error occurred in the cluster
>>>>>>>>>>> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
>>>>>>>>>>> Application failed unexpectedly.
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>>>>>>>>>>>
>>>>>>>>>>>  org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>> Caused by: java.util.concurrent.CancellationException: null
>>>>>>>>>>>  java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\t...
>>>>>>>>>>> 22 common frames omitted"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.827+01:00","@version":"1","message":"Shutting
>>>>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status
>>>>>>>>>>> UNKNOWN. Diagnostics Cluster entrypoint has been closed
>>>>>>>>>>> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.915+01:00","@version":"1","message":"Stopped
>>>>>>>>>>> BLOB server at 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.828+01:00","@version":"1","message":"Shutting
>>>>>>>>>>> down rest
>>>>>>>>>>> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
>>>>>>>>>>> shutdown hook","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.930+01:00","@version":"1","message":"Removing
>>>>>>>>>>> cache directory
>>>>>>>>>>> /tmp/flink-web-6c82da46-786b-445b-b776-dce60b7e738d/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Closing
>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.931+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@673e76b3
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Loop
>>>>>>>>>>> thread
>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5eff83cc
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Shut
>>>>>>>>>>> down
>>>>>>>>>>> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Closing
>>>>>>>>>>> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.932+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7069f076
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.933+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50387663
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.934+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@41bfa9e9
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5407e8bc
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.935+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:22.936+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Closing
>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.015+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5ef85555
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Loop
>>>>>>>>>>> thread
>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@11057c58
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-31","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.016+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-31","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.017+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Closing
>>>>>>>>>>> the
>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Suspending
>>>>>>>>>>> the
>>>>>>>>>>> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.018+01:00","@version":"1","message":"Stopping
>>>>>>>>>>> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Force
>>>>>>>>>>> closing the watch
>>>>>>>>>>> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@5287b20b
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Loop
>>>>>>>>>>> thread
>>>>>>>>>>> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>>>>>>>>>>>
>>>>>>>>>>>  io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>>>>
>>>>>>>>>>>  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>>>>  java.lang.Thread.run(Thread.java:748)"}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"The
>>>>>>>>>>> watcher is
>>>>>>>>>>> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-25","level":"INFO","level_value":20000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.019+01:00","@version":"1","message":"Closing
>>>>>>>>>>> websocket
>>>>>>>>>>> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@6bcacc81
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-25","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.020+01:00","@version":"1","message":"WebSocket
>>>>>>>>>>> close received. code: 1000, reason:
>>>>>>>>>>> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>> {"@timestamp":"2021-11-29T06:03:23.021+01:00","@version":"1","message":"Ignoring
>>>>>>>>>>> onClose for already closed/closing
>>>>>>>>>>> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
>>>>>>>>>>> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>