Re: [I] Feature Request - Pool Support for Task Groups in Apache Airflow [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk commented on issue #35689:
URL: https://github.com/apache/airflow/issues/35689#issuecomment-1952982168

   One more comment here (result of discussion in Slack).
   
   I think it would be indeed a very nice feature to add in Airflow, but likely you can achieve it by using some external DB (might be any DB - relational or not) whre you can implement transaction doing an atomic "Increase or Wait" functionality - similar to what semaphores are doing in proces synchronisaton, and using the new Setup/Teardown functionality in Airflow.
   
   All you need is to have a call (might be an SQL QUERY)) that will atomically run "Increase or Wait" call (I.e. attempt to increase some value but when it reaches some threashold - wait and attmpt again - either in a loop or when you are notified that the counter/semaphore has been decreased).
   
   For each of your group - you should write a custom Setup task doing "Increase or  wait" kind of functionality, and Teardown task doing "decrease". That should achieve exactly what you want - each task group would continue only after Setup manages to successfully increase such external semaphore. 
   
   This is of course not super easy to implement and would be nice to have similar feature in Airflow, but it's entirely possible to implement something like that with - I think any remote transaction DB you would want to use, or even with object storage most likely. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org