[jira] [Commented] (AIRFLOW-5931) Spawning new python interpreter for every task slow

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127549#comment-17127549 ] 

ASF GitHub Bot commented on AIRFLOW-5931:
-----------------------------------------

mik-laj commented on pull request #6627:
URL: https://github.com/apache/airflow/pull/6627#issuecomment-640178886


   If anyone has problems with this change, they can use this code to disable this change.
   ```python
   from airflow.task.task_runner import standard_task_runner
   standard_task_runner.CAN_FORK=False
   
   import os
   import argcomplete
   from airflow.configuration import conf
   from airflow.bin.cli import CLIFactory
   if __name__ == '__main__':
       if conf.get("core", "security") == 'kerberos':
           os.environ['KRB5CCNAME'] = conf.get('kerberos', 'ccache')
           os.environ['KRB5_KTNAME'] = conf.get('kerberos', 'keytab')
       parser = CLIFactory.get_parser()
       argcomplete.autocomplete(parser)
       args = parser.parse_args()
       args.func(args)
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Spawning new python interpreter for every task slow
> ---------------------------------------------------
>
>                 Key: AIRFLOW-5931
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5931
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executors, worker
>    Affects Versions: 2.0.0
>            Reporter: Ash Berlin-Taylor
>            Assignee: Ash Berlin-Taylor
>            Priority: Major
>             Fix For: 1.10.7
>
>
> There are a number of places in the Executors and Task Runners where we spawn a whole new python interpreter.
> My profiling has shown that this is slow. Rather than running a fresh python interpreter which then has to re-load all of Airflow and its dependencies we should use {{os.fork}} when it is available/suitable which should speed up task running, espeically for short lived tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)