You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/05/14 07:16:00 UTC
[jira] [Commented] (AIRFLOW-5931) Spawning new python interpreter
for every task slow
[ https://issues.apache.org/jira/browse/AIRFLOW-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106997#comment-17106997 ]
ASF GitHub Bot commented on AIRFLOW-5931:
-----------------------------------------
davido912 commented on pull request #6627:
URL: https://github.com/apache/airflow/pull/6627#issuecomment-628438830
@ewjmulder could you please elaborate a bit more in regards to what did you actually add to the log_config file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Spawning new python interpreter for every task slow
> ---------------------------------------------------
>
> Key: AIRFLOW-5931
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5931
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executors, worker
> Affects Versions: 2.0.0
> Reporter: Ash Berlin-Taylor
> Assignee: Ash Berlin-Taylor
> Priority: Major
> Fix For: 1.10.7
>
>
> There are a number of places in the Executors and Task Runners where we spawn a whole new python interpreter.
> My profiling has shown that this is slow. Rather than running a fresh python interpreter which then has to re-load all of Airflow and its dependencies we should use {{os.fork}} when it is available/suitable which should speed up task running, espeically for short lived tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)