[jira] [Updated] (AIRFLOW-6361) Run LocalTaskJob directly in Celery task

["Kamil Bregula (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kamil Bregula updated AIRFLOW-6361:
-----------------------------------
    Labels: performance  (was: )

> Run LocalTaskJob directly in Celery task
> ----------------------------------------
>
>                 Key: AIRFLOW-6361
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6361
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executors
>    Affects Versions: 1.10.6
>            Reporter: Kamil Bregula
>            Priority: Major
>              Labels: performance
>
> Hello,
> Celery runs the CLI first command, which contains LocalTaskJob. LocalTaskJob is responsible for starting the next user-code process. This level of isolation is redundant because LocalTaskJob doesn't execute unsafe code. The first command is run by a new process creation, not by a fork, so this is an expensive operation.
> According to preliminary measurements, this change results in an increase in performance close to 30%.
> I will provide more information in PR.
> Best regards
> Kamil Bregula
> After:
> ```
> real	0m38.394s
> user	0m4.340s
> sys	0m1.600s
> real	0m38.355s
> user	0m4.700s
> sys	0m1.340s
> real	0m38.675s
> user	0m4.760s
> sys	0m1.530s
> real	0m38.488s
> user	0m4.770s
> sys	0m1.280s
> real	0m38.434s
> user	0m4.600s
> sys	0m1.390s
> real	0m38.378s
> user	0m4.500s
> sys	0m1.270s
> real	0m38.106s
> user	0m4.200s
> sys	0m1.100s
> real	0m38.082s
> user	0m4.170s
> sys	0m1.030s
> real	0m38.173s
> user	0m4.290s
> sys	0m1.340s
> real	0m38.161s
> user	0m4.460s
> sys	0m1.370s	
> ```
> Before:
> ```
> real	0m53.488s
> user	0m5.140s
> sys	0m1.700s
> real	1m8.288s
> user	0m6.430s
> sys	0m2.200s
> real	0m53.371s
> user	0m5.330s
> sys	0m1.630s
> real	0m58.939s
> user	0m6.470s
> sys	0m1.730s
> real	0m53.255s
> user	0m4.950s
> sys	0m1.640s
> real	0m58.802s
> user	0m5.970s
> sys	0m1.790s
> real	0m58.449s
> user	0m5.380s
> sys	0m1.580s
> real	0m53.308s
> user	0m5.120s
> sys	0m1.430s
> real	0m53.485s
> user	0m5.220s
> sys	0m1.290s
> real	0m53.387s
> user	0m5.020s
> sys	0m1.590s
> ```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)