[jira] [Created] (FLINK-15843) Do not violently kill TaskManagers

["Canbin Zheng (Jira)" <ji...@apache.org>]

Canbin Zheng created FLINK-15843:
------------------------------------

             Summary: Do not violently kill TaskManagers
                 Key: FLINK-15843
                 URL: https://issues.apache.org/jira/browse/FLINK-15843
             Project: Flink
          Issue Type: Sub-task
          Components: Deployment / Kubernetes
    Affects Versions: 1.10.0
            Reporter: Canbin Zheng
             Fix For: 1.11.0


The current solution of stopping a TaskManager instance when JobManager sends a deletion request is by directly calling ${\{KubernetesClient.pods().withName().delete}}, thus that instance would be violently killed with a _KILL_ signal and having no chance to clean up, which could cause problems because we expect the process to gracefully terminate when it is no longer needed.

Refer to the guide of [Termination of Pods|[https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods]|https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods],], we know that on Kubernetes a _TERM_ signal would be first sent to the main process in each container, and may be followed up with a force _KILL_ signal if the grace period has expired; the Unix signal will be sent to the process which has PID 1 ([Docker Kill|https://docs.docker.com/engine/reference/commandline/kill/]), however, the TaskManagerRunner Process is spawned by {color:#172b4d}/opt/flink/bin/kubernetes-entry.sh {color}and could never have PID 1, so it would never receive the Unix signal_._

 

One walk around could be that JobManager firstly sends a *KILL_WORKER* message to the  TaskManager, then the TaskManager gracefully terminates itself to ensure that the clean-up is completely finished, lastly, the JobManager deletes the Pod after a configurable graceful shut-down period.

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)