[GitHub] [airflow] ginevragaudioso commented on issue #13975: Allow multiple pools for one task

[GitBox <gi...@apache.org>]

ginevragaudioso commented on issue #13975:
URL: https://github.com/apache/airflow/issues/13975#issuecomment-771785185


   Thanks @anushkrishnav ! I can't provide much guidance on the implementation side of things, as I stated before I did not take the time to look into airflow source code deep enough to be able to give any meaningful advice. All I know is that it seems that the one-pool assumption goes all the way into SQL, so this may require a table model change, but I could be wrong here.
   In terms of feature request, I can help clarify the desired outcome. The goal is to be able to call something like (`BashOperator` is only an example, but this change would need to go into `BaseOperator` so that all operators inherit it):
   `
   my_task = BashOperator(
       # some kwargs here, unrelated to pool(s)
       pools=[RESOURCE_A_POOL, RESOURCE_B_POOL],
       pools_slots=[1, 1], # this could be default, but just mentioning here since this kwarg also needs to change
   )
   `
   Currently, the `pools` argument that takes a list of pools does not exist, we only have a `pool` argument that takes one pool.
   With the proposed feature in this issue, and the proposed new call in this comment, `my_task` can only be scheduled when there is one slot available in all declared pools, and `my_task` will take up one slot in all pools. I think it's ok to leave resource ordering (which is needed to avoid deadlocks probably) to the user, and airflow could just "book" the pools in the order they are given.
   
   Please let me know if you need any more guidance from me, and thanks for replying to this issue!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org