You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "g-dailey (via GitHub)" <gi...@apache.org> on 2026/05/11 19:55:04 UTC

[I] Document that backfill DAG runs are deprioritized over scheduled and manual runs in the scheduler loop [airflow]

g-dailey opened a new issue, #66729:
URL: https://github.com/apache/airflow/issues/66729

   ### Description
   
   Airflow's scheduler deliberately deprioritizes backfill DAG runs relative to regular scheduled and manual runs within each scheduling cycle. This behavior is intentional and prevents backfills from starving production pipelines, but it is not documented anywhere in the official docs, leading to user confusion.
   
   What's happening under the hood: 
   The scheduler uses `max_dagruns_per_loop_to_schedule` (default: 20) to limit how many DAG runs it examines per cycle. Within that batch, DAG runs are ordered by:
   `ORDER BY backfill_dag_run.sort_ordinal ASC NULLS FIRST`
   
   Non-backfill runs have no row in `backfill_dag_run`, so the `LEFT JOIN` produces `NULL` for `sort_ordinal`. `NULLS FIRST` places them at the top of the list. Backfill runs, which have actual `sort_ordinal` values, always sort after all regular runs.
   
   ### Use case/motivation
   
   On the backfill dagrun page, it will be helpful to have a short blurb letting user know that the backfill run will have lower priority over the scheduled and manual runs, which can lead dagrun remaining in scheduled state for an extended period of time.
   
   <img width="2180" height="206" alt="Image" src="https://github.com/user-attachments/assets/f13eb5f9-a4f4-43fb-ab7e-44052efc43de" />
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Document that backfill DAG runs are deprioritized over scheduled and manual runs in the scheduler loop [airflow]

Posted by "nnguyen168 (via GitHub)" <gi...@apache.org>.
nnguyen168 commented on issue #66729:
URL: https://github.com/apache/airflow/issues/66729#issuecomment-4516748560

   Hello @g-dailey , I'd like to contribute to this issue. Could you please tell me whether this is still open, and assign me to it? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Document that backfill DAG runs are deprioritized over scheduled and manual runs in the scheduler loop [airflow]

Posted by "choo121600 (via GitHub)" <gi...@apache.org>.
choo121600 closed issue #66729: Document that backfill DAG runs are deprioritized over scheduled and manual runs in the scheduler loop
URL: https://github.com/apache/airflow/issues/66729


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org