[GitHub] [airflow] fjetter commented on issue #16739: Queue support for DaskExecutor using Dask Worker Resources

[GitBox <gi...@apache.org>]

fjetter commented on issue #16739:
URL: https://github.com/apache/airflow/issues/16739#issuecomment-874578147


   I'm not familiar enough with the queue functionality of airflow to know what the expected behaviour should be. In dask we have broadly speaking two ti three mechanism to limit concurrency on task level and/or control assignments to workers.
   
   If you want to limit the number of assigned tasks, i.e. want to ensure that tasks are not yet assigned to a worker before it is allowed to be executed, resources are the way to go. 
   
   If you want to control which workers are allowed to work on a given task, the `workers` keyword might be a better fit but that doesn't control concurrency (other than the intrinsic limit a single worker exposes)
   
   If you want to ensure that only a limited number of tasks is executed but it is fine for them to be assigned to a worker and may even block a worker, we have a [Semaphore](https://distributed.dask.org/en/latest/api.html#distributed.Semaphore) which could be used. 
   
   Which is the best to pick depends on how queuing in airflow is supposed to work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org