You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Ash Berlin-Taylor (Jira)" <ji...@apache.org> on 2019/11/14 13:47:00 UTC

[jira] [Created] (AIRFLOW-5931) Spawning new python interpreter is slow

Ash Berlin-Taylor created AIRFLOW-5931:
------------------------------------------

             Summary: Spawning new python interpreter is slow
                 Key: AIRFLOW-5931
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5931
             Project: Apache Airflow
          Issue Type: Improvement
          Components: executors, worker
    Affects Versions: 2.0.0
            Reporter: Ash Berlin-Taylor
            Assignee: Ash Berlin-Taylor


There are a number of places in the Executors and Task Runners where we spawn a whole new python interpreter.

My profiling has shown that this is slow. Rather than running a fresh python interpreter which then has to re-load all of Airflow and its dependencies we should use {{os.fork}} when it is available/suitable which should speed up task running, espeically for short lived tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)