[jira] [Created] (AIRFLOW-6361) Run LocalTaskJob directly in Celery task

["Kamil Bregula (Jira)" <ji...@apache.org>]

Kamil Bregula created AIRFLOW-6361:
--------------------------------------

             Summary: Run LocalTaskJob directly in Celery task
                 Key: AIRFLOW-6361
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6361
             Project: Apache Airflow
          Issue Type: Improvement
          Components: executors
    Affects Versions: 1.10.6
            Reporter: Kamil Bregula


Hello,

Celery runs the CLI first command, which contains LocalTaskJob. LocalTaskJob is responsible for starting the next user-code process. This level of isolation is redundant because LocalTaskJob doesn't execute unsafe code. The first command is run by a new process creation, not by a fork, so this is an expensive operation.

According to preliminary measurements, this change results in an increase in performance close to 30%.

I will provide more information in PR.

Best regards
Kamil Bregula

After:
```
real	0m38.394s
user	0m4.340s
sys	0m1.600s

real	0m38.355s
user	0m4.700s
sys	0m1.340s

real	0m38.675s
user	0m4.760s
sys	0m1.530s

real	0m38.488s
user	0m4.770s
sys	0m1.280s

real	0m38.434s
user	0m4.600s
sys	0m1.390s

real	0m38.378s
user	0m4.500s
sys	0m1.270s

real	0m38.106s
user	0m4.200s
sys	0m1.100s

real	0m38.082s
user	0m4.170s
sys	0m1.030s

real	0m38.173s
user	0m4.290s
sys	0m1.340s

real	0m38.161s
user	0m4.460s
sys	0m1.370s	
```

Before:
```
real	0m53.488s
user	0m5.140s
sys	0m1.700s

real	1m8.288s
user	0m6.430s
sys	0m2.200s

real	0m53.371s
user	0m5.330s
sys	0m1.630s

real	0m58.939s
user	0m6.470s
sys	0m1.730s

real	0m53.255s
user	0m4.950s
sys	0m1.640s

real	0m58.802s
user	0m5.970s
sys	0m1.790s

real	0m58.449s
user	0m5.380s
sys	0m1.580s

real	0m53.308s
user	0m5.120s
sys	0m1.430s

real	0m53.485s
user	0m5.220s
sys	0m1.290s

real	0m53.387s
user	0m5.020s
sys	0m1.590s
```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)