[GitHub] [airflow] zachary-povey commented on issue #13975: Allow multiple pools for one task

[GitBox <gi...@apache.org>]

zachary-povey commented on issue #13975:
URL: https://github.com/apache/airflow/issues/13975#issuecomment-806538498


   There is another fundamental issue with pools in airflow that this would help solve: if you use pools to limit concurrency across different DAGs you [lose the ability to prioritise one task above another  in the same DAG](https://stackoverflow.com/questions/66779968/how-does-task-priority-work-in-airflow-when-both-pools-and-dag-concurrency-are-u), which may have a concurrency limit on it. This is because the `priority_weight` argument for the operator refers to priority within a pool not a DAG.
   
   If we could add tasks to more than one pool, you could then have each task in a pool for the DAG and another pool for limiting concurrency with a task in seperate DAG which shares a resource.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org