[GitHub] [airflow] fj-sanchez commented on pull request #16358: Only create dagruns when max_active_runs is not reached

[GitBox <gi...@apache.org>]

fj-sanchez commented on pull request #16358:
URL: https://github.com/apache/airflow/pull/16358#issuecomment-859459745


   > > > > Does this PR literally restrict from staggering new dagruns of there's already running one?
   > > > 
   > > > 
   > > > I don't think I understand you correctly but if you mean creating more dagruns when the max_active_runs have been reached, then yes. You can't create more runs if the max_active_runs has been reached
   > > 
   > > 
   > > I would typically expect that making more dagruns would imply their sequential execution and queuing one after another. In this case, it would straight not allow it to queue them. Removing feature that people looking for doesn't sound like a solution for initial problem we have here. Afterall, why would people want to ever trigger dag multiple times, if they want only one dagrun be present at a time... Because they want to add another run to queue! The only consequence if this gets merged would be more people turning to doing poll as workaround (spam trigger until it actually is successful)
   > 
   > I don't think you understand the change. You can increase the number of dagruns. It's controlled by the max_active_runs in a dag. The default is 16 and you can increase it as well.
   
   I agree with @dimon222, this isn't addressing the problem that we have. I have an external system triggering triggering dag_runs (parameterised using different `conf`), I would expect that Airflow, as an orchestrator, would enqueue requests when it's already at `max_active_runs` capacity, not to refuse any further external request to run a dag.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org