[jira] [Resolved] (AIRFLOW-5931) Spawning new python interpreter for every task slow

["Ash Berlin-Taylor (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ash Berlin-Taylor resolved AIRFLOW-5931.
----------------------------------------
    Fix Version/s: 1.10.7
       Resolution: Fixed

> Spawning new python interpreter for every task slow
> ---------------------------------------------------
>
>                 Key: AIRFLOW-5931
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5931
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executors, worker
>    Affects Versions: 2.0.0
>            Reporter: Ash Berlin-Taylor
>            Assignee: Ash Berlin-Taylor
>            Priority: Major
>             Fix For: 1.10.7
>
>
> There are a number of places in the Executors and Task Runners where we spawn a whole new python interpreter.
> My profiling has shown that this is slow. Rather than running a fresh python interpreter which then has to re-load all of Airflow and its dependencies we should use {{os.fork}} when it is available/suitable which should speed up task running, espeically for short lived tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)