You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/10/01 20:35:00 UTC
[jira] [Commented] (AIRFLOW-6361) Run LocalTaskJob directly in
Celery task
[ https://issues.apache.org/jira/browse/AIRFLOW-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17205807#comment-17205807 ]
ASF GitHub Bot commented on AIRFLOW-6361:
-----------------------------------------
dimberman commented on pull request #6905:
URL: https://github.com/apache/airflow/pull/6905#issuecomment-702382201
@mik-laj are we still pursuing this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Run LocalTaskJob directly in Celery task
> ----------------------------------------
>
> Key: AIRFLOW-6361
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6361
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executors
> Affects Versions: 1.10.6
> Reporter: Kamil Bregula
> Priority: Major
> Labels: performance
>
> Hello,
> Celery runs the CLI first command, which contains LocalTaskJob. LocalTaskJob is responsible for starting the next user-code process. This level of isolation is redundant because LocalTaskJob doesn't execute unsafe code. The first command is run by a new process creation, not by a fork, so this is an expensive operation.
> According to preliminary measurements, this change results in an increase in performance close to 30%.
> I will provide more information in PR.
> Best regards
> Kamil Bregula
> After:
> ```
> real 0m38.394s
> user 0m4.340s
> sys 0m1.600s
> real 0m38.355s
> user 0m4.700s
> sys 0m1.340s
> real 0m38.675s
> user 0m4.760s
> sys 0m1.530s
> real 0m38.488s
> user 0m4.770s
> sys 0m1.280s
> real 0m38.434s
> user 0m4.600s
> sys 0m1.390s
> real 0m38.378s
> user 0m4.500s
> sys 0m1.270s
> real 0m38.106s
> user 0m4.200s
> sys 0m1.100s
> real 0m38.082s
> user 0m4.170s
> sys 0m1.030s
> real 0m38.173s
> user 0m4.290s
> sys 0m1.340s
> real 0m38.161s
> user 0m4.460s
> sys 0m1.370s
> ```
> Before:
> ```
> real 0m53.488s
> user 0m5.140s
> sys 0m1.700s
> real 1m8.288s
> user 0m6.430s
> sys 0m2.200s
> real 0m53.371s
> user 0m5.330s
> sys 0m1.630s
> real 0m58.939s
> user 0m6.470s
> sys 0m1.730s
> real 0m53.255s
> user 0m4.950s
> sys 0m1.640s
> real 0m58.802s
> user 0m5.970s
> sys 0m1.790s
> real 0m58.449s
> user 0m5.380s
> sys 0m1.580s
> real 0m53.308s
> user 0m5.120s
> sys 0m1.430s
> real 0m53.485s
> user 0m5.220s
> sys 0m1.290s
> real 0m53.387s
> user 0m5.020s
> sys 0m1.590s
> ```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)