[jira] [Comment Edited] (FLINK-17176) Slow down Pod re-creation in KubernetesResourceManager#PodCallbackHandler

["Canbin Zheng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-17176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085383#comment-17085383 ] 

Canbin Zheng edited comment on FLINK-17176 at 4/17/20, 3:08 AM:
----------------------------------------------------------------

Of course, they should share the same retry logic. I think you are right, this ticket could be a subtask of {color:#4a6ee0}FLINK-17127{color}


was (Author: felixzheng):
Of course, they should share the same retry logic, and this ticket could be a subtask of {color:#4a6ee0}FLINK-17127{color}

> Slow down Pod re-creation in KubernetesResourceManager#PodCallbackHandler
> -------------------------------------------------------------------------
>
>                 Key: FLINK-17176
>                 URL: https://issues.apache.org/jira/browse/FLINK-17176
>             Project: Flink
>          Issue Type: Improvement
>          Components: Deployment / Kubernetes
>    Affects Versions: 1.10.0
>            Reporter: Canbin Zheng
>            Priority: Major
>             Fix For: 1.11.0
>
>
> In the native K8s setups, there are some cases that we do not control the speed of pod re-creation which poses potential risks to flood the K8s API Server in the {{PodCallbackHandler}} implementation of {{KubernetesResourceManager.}}
> Here are steps to reproduce this kind of problems:
>  # Mount the {{/opt/flink/log}} in the Container of TaskManager to a path on the K8s nodes via HostPath, make sure that the path exists but the TaskManager process has no write permission. We can achieve this via the [user-specified pod template support|https://issues.apache.org/jira/browse/FLINK-15656] or just hardcode it for testing only.
>  # Launch a session cluster
>  # Submit a new job to the session cluster, as expected, we can observe that the Pod constantly fails quickly during launching the main Container, then the {{KubernetesResourceManager#onModified}} is invoked to re-create a new Pod immediately, without any speed control.
> To sum up, once the {{KubernetesResourceManager}} receives the Pod *ADD* event and that Pod is terminated before successfully registering into the {{KubernetesResourceManager}}, the {{KubernetesResourceManager}} will send another creation request to K8s API Server immediately.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)