[GitHub] [airflow] dimberman commented on issue #5788: [POC] multi-threading using asyncio

[GitBox <gi...@apache.org>]

dimberman commented on issue #5788: [POC] multi-threading using asyncio
URL: https://github.com/apache/airflow/pull/5788#issuecomment-570361064
 
 
   @a
   
   > > @ash recently made some changes to the LocalExecutor to use os.fork instead of processes, this makes the multithreading in the LocalExecutor much faster and lighter weight.
   > 
   > That's curious. AFAIK, `os.fork` is a child process, not a thread. In python, the GIL effectively makes all threading run with preemptive scheduling as though it is running in a single thread and the performance gains are almost none and sometimes it can hurt performance because the system threads start thrashing to get a hold of the GIL. The threads do not run in parallel because of the GIL, so parallel performance requires multiprocessing with a process pool. I recently delved into a good article on this topic, with a lot of good references for more details:
   > 
   > * https://realpython.com/async-io-python
   > * https://realpython.com/python-gil/
   > * plus links to talks by David Beazley and the gilectomy etc.
   
   @ashb perhaps you could speak to this better than I can?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services