[GitHub] [airflow] ephraimbuddy commented on pull request #21413: Fix max_active_runs=1 not scheduling runs when min_file_process_interval is high

[GitBox <gi...@apache.org>]

ephraimbuddy commented on pull request #21413:
URL: https://github.com/apache/airflow/pull/21413#issuecomment-1033098785


   It turned out that we just needed to flush the session at this point and not that dag.get_num_active_runs is giving a wrong value.
   
   I'm wondering if it's still necessary to run dag_run.schedule_tis for a finished dag_run?
   I'm proposing to change the current code in that area to this:
   ```python
   if dag_run.state in State.finished:
         session.flush()  # to update the dag_run state
         active_runs = dag.get_num_active_runs(only_running=False, session=session)
         # Work out if we should allow creating a new DagRun now?
         if self._should_update_dag_next_dagruns(dag, dag_model, active_runs):
             dag_model.calculate_dagrun_date_fields(dag, dag.get_run_data_interval(dag_run))
   else:
       # This will do one query per dag run. We "could" build up a complex
       # query to update all the TIs across all the execution dates and dag
       # IDs in a single query, but it turns out that can be _very very slow_
       # see #11147/commit ee90807ac for more details
       dag_run.schedule_tis(schedulable_tis, session)
   return callback
   ```
   cc: @ashb @potiuk @uranusjr @jedcunningham 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org