[GitHub] [airflow] dimberman commented on issue #5788: [POC] multi-threading using asyncio

[GitBox <gi...@apache.org>]

dimberman commented on issue #5788: [POC] multi-threading using asyncio
URL: https://github.com/apache/airflow/pull/5788#issuecomment-569744458
 
 
   Is this necessary?
   
   Asyncio doesn’t do multi-threading. It uses a single thread but switches off what coroutine uses that thread based on readiness. This is actually a pretty important distinction when it comes to parallel processing in python. @ash recently made some changes to the LocalExecutor to use os.fork instead of processes, this makes the multithreading in the LocalExecutor much faster and lighter weight.
   
   I don’t think that there’s really that much blocking in our operators outside of specific ones that are making SQL calls etc. I don’t think this would be more performant than the current LocalExecutor but would be glad to discuss potential benefits I’m not seeing/if you’ve done speed comparisons.
   
   via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2]
   On Sun, Dec 29, 2019 at 5:56 PM, Darren Weber <no...@github.com> wrote:
   Tagging this as related to
   
    * https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-28%3A+Add+AsyncExecutor+option [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-28%3A+Add+AsyncExecutor+option]
    * https://issues.apache.org/jira/browse/AIRFLOW-6395 [https://issues.apache.org/jira/browse/AIRFLOW-6395]
   
   —
   You are receiving this because you authored the thread.
   Reply to this email directly, view it on GitHub [https://github.com/apache/airflow/pull/5788?email_source=notifications&email_token=AAUFRAX635KVKD3QBFCES5DQ3FIONA5CNFSM4ILDNB72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHZNMGY#issuecomment-569562651] , or unsubscribe [https://github.com/notifications/unsubscribe-auth/AAUFRAVWOHYCRQB6N456YUTQ3FIONANCNFSM4ILDNB7Q] .

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services