[GitHub] [airflow] dazza-codes edited a comment on issue #6984: [WIP][AIRFLOW-6395] Add an asyncio compatible dask executor [AIP-28]

[GitBox <gi...@apache.org>]

dazza-codes edited a comment on issue #6984: [WIP][AIRFLOW-6395] Add an asyncio compatible dask executor [AIP-28]
URL: https://github.com/apache/airflow/pull/6984#issuecomment-609060255
 
 
   At some point, I hope to circle back to asyncio in Airflow.  For now, I'm working more on the low-level details in another project, https://gitlab.com/dazza-codes/aio-aws, so that I can avoid some complexity of the legacy work in Airflow and any conflicts with the larger community use-cases.  There's a lot of details in the asyncio approach that need to be resolved before I could confidently contribute something to Airflow.
   
   With regard to dask and async executors, dask already has the right backend to support async execution, so this should be low-hanging fruit.  I apologise for neglecting it for a bit.  I did explore some basic elements of an async executor in
   - https://gitlab.com/dazza-codes/python-notes/-/blob/master/notes/async_executor.py
   - https://gitlab.com/dazza-codes/python-notes/-/blob/master/tests/test_async_executor.py
   
   I'm not entirely convinced that using a thread for the loop and passing coroutines across a thread-safe bridge is the right way to go.  For instance, it's noted in https://docs.python.org/3.6/library/asyncio-dev.html#concurrency-and-multithreading that `Most asyncio objects are not thread safe.`
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services