You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Yingbo Wang (Jira)" <ji...@apache.org> on 2020/02/22 01:05:00 UTC

[jira] [Created] (AIRFLOW-6874) There are risks that subprocesses not killed when a task failed

Yingbo Wang created AIRFLOW-6874:
------------------------------------

             Summary: There are risks that subprocesses not killed when a task failed
                 Key: AIRFLOW-6874
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6874
             Project: Apache Airflow
          Issue Type: Improvement
          Components: worker
    Affects Versions: 1.10.4
            Reporter: Yingbo Wang
            Assignee: Yingbo Wang


Many airflow tasks create subprocesses and these subprocesses may create more subprocesses. In our experience, there is a risk that although a task failed and tried to reap the process group, there are still left over processes running and cause issues with both resources and correctness.

Propose to improve the cgroup task runner to reap all processes for current node on node termination. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)