Re: [PR] Fix dagrun starvation [airflow]

["uranusjr (via GitHub)" <gi...@apache.org>]

uranusjr commented on code in PR #64109:
URL: https://github.com/apache/airflow/pull/64109#discussion_r3744833783


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -808,16 +808,33 @@ def get_queued_dag_runs_to_set_running(cls, session: Session) -> ScalarResult[Da
             .subquery()
         )
 
-        query = (
-            select(cls)
-            .where(cls.state == DagRunState.QUEUED)
+        available_dagruns_rn = (
+            select(
+                DagRun.dag_id,
+                DagRun.id,
+                running_drs.c.num_running,
+                func.row_number()
+                .over(
+                    partition_by=[DagRun.dag_id, DagRun.backfill_id],
+                    order_by=[
+                        nulls_first(cast("ColumnElement[Any]", BackfillDagRun.sort_ordinal), session=session),
+                        nulls_first(
+                            cast("ColumnElement[Any]", cls.last_scheduling_decision), session=session
+                        ),
+                        nulls_first(running_drs.c.num_running, session=session),
+                        DagRun.run_after,
+                    ],
+                )
+                .label("rn"),
+            )
+            .where(DagRun.state == DagRunState.QUEUED)
             .join(
-                DagModel,
+                running_drs,
                 and_(
-                    DagModel.dag_id == cls.dag_id,
-                    DagModel.is_paused == false(),
-                    DagModel.is_stale == false(),
+                    running_drs.c.dag_id == DagRun.dag_id,
+                    running_drs.c.backfill_id == DagRun.backfill_id,

Review Comment:
   This needs null protection, otherwise we won’t ever be able to execute a non-backfill dagrun because `NULL == NULL` evaluates to `FALSE` in SQL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org