Re: [I] Feature Request - Pool Support for Task Groups in Apache Airflow [airflow]

["bruno-uy (via GitHub)" <gi...@apache.org>]

bruno-uy commented on issue #35689:
URL: https://github.com/apache/airflow/issues/35689#issuecomment-2152932034

   I'm trying to coordinate two groups of tasks in different pipelines and this feature would be really valuable.
   
   For my use case it would be enough to have a `pool` in the `TaskGroup` level that works as the normal `pool` in a task.
   
   Let me give an example of the problem with using that in a single task:
   ```
   Pipeline A:
   t1 >> TaskGroup(t2, t3) >> t4
   
   Pipeline B:
   t5 >> TaskGroup(t6, t7) >> t8
   ```
   
   If I add the `pool` to `t2`, `t3`, `t6` and `t7` (let's say it's only 1 slot), the first one to start will take the spot. Let's say it's `t2`. If `t6` is ready to run before `t3`, `t6` will get the spot on the pool, but I want the whole task group to run when they acquire the pool.
   
   Hope this example helps. Is this similar to what you had in mind @rodolfo-nobrega?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org