[GitHub] [airflow] ashb commented on a change in pull request #17945: Fix max_active_runs not allowing scheduling of other dagruns

[GitBox <gi...@apache.org>]

ashb commented on a change in pull request #17945:
URL: https://github.com/apache/airflow/pull/17945#discussion_r700941549



##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -970,48 +971,61 @@ def _start_queued_dagruns(
         session: Session,
     ) -> int:
         """Find DagRuns in queued state and decide moving them to running state"""
-        dag_runs = self._get_next_dagruns_to_examine(State.QUEUED, session)
-
-        active_runs_of_dags = defaultdict(
-            lambda: 0,
-            session.query(DagRun.dag_id, func.count('*'))
-            .filter(  # We use `list` here because SQLA doesn't accept a set
-                # We use set to avoid duplicate dag_ids
-                DagRun.dag_id.in_(list({dr.dag_id for dr in dag_runs})),
-                DagRun.state == State.RUNNING,
-            )
-            .group_by(DagRun.dag_id)
-            .all(),
+        active_dags = (
+            session.query(DM)
+            .filter(DM.is_active == expression.true(), DM.is_paused == expression.false())
+            .all()
         )
+        max_number = conf.getint('scheduler', 'max_dagruns_per_loop_to_schedule', fallback=20)
+        # There's a possibility of exceeding this max_number or going below it as a result of the logic below
+        if len(active_dags) > max_number:
+            max_number = 1

Review comment:
       This isn't the right behaviour -- there could be 1000 "active" dags, but if they don't have any active dag runs then they shouldn't affect scheduling.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org