[jira] [Commented] (FLINK-17177) Handle ERROR event correctly in KubernetesResourceManager#onError

["Yang Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-17177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085378#comment-17085378 ] 

Yang Wang commented on FLINK-17177:
-----------------------------------

Hi [~felixzheng], thanks for creating this insightful tickets. I am not sure whether we need to create a new watcher here since the {{WatchConnectionManager}} in fabric8 kubernetes client has internal retry logics for http/websocket failure.

Another concern is if the reconnect limit exhausts, may because of pod network or K8s api server down, i think we need to throw fatal error and exit the jobmanager pod. Then a new jobmanager pod will be started to take over. This is also the logic for YARN deployment. If the {{AMRMClient}} heartbeats with YARN ResourceManager failed(the ipc client has retried enough times), then {{YarnResourceManager}} will also call the onFatalError.

> Handle ERROR event correctly in KubernetesResourceManager#onError
> -----------------------------------------------------------------
>
>                 Key: FLINK-17177
>                 URL: https://issues.apache.org/jira/browse/FLINK-17177
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / Kubernetes
>    Affects Versions: 1.10.0, 1.10.1
>            Reporter: Canbin Zheng
>            Priority: Major
>             Fix For: 1.11.0
>
>
> Currently, once we receive an *ERROR* event that is sent from the K8s API server via the K8s {{Watcher}}, then {{KubernetesResourceManager#onError}} will handle it by calling the {{KubernetesResourceManager#removePodIfTerminated}}. This may be incorrect since the *ERROR* event indicates an exception in the HTTP layer that is caused by the K8s Server, which means the previously created {{Watcher}} may be no longer available and we'd better re-create the {{Watcher}} immediately.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)