Re: [PR] Fix backfill marked complete before DagRuns are created [airflow]

["Copilot (via GitHub)" <gi...@apache.org>]

Copilot commented on code in PR #62561:
URL: https://github.com/apache/airflow/pull/62561#discussion_r2926779967


##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1981,6 +1981,9 @@ def _mark_backfills_complete(self, session: Session = NEW_SESSION) -> None:
         # todo: AIP-78 simplify this function to an update statement
         query = select(Backfill).where(
             Backfill.completed_at.is_(None),
+            # Guard: backfill must have at least one association,
+            # otherwise it is still being set up (see #61375).
+            exists(select(BackfillDagRun.id).where(BackfillDagRun.backfill_id == Backfill.id)),

Review Comment:
   The new `EXISTS(backfill_dag_run)` guard means a Backfill that gets committed but never manages to create any `BackfillDagRun` rows (e.g. if `_create_backfill()` errors/crashes after the `session.commit()` at `airflow/models/backfill.py:605`) will never be auto-completed by the scheduler. Since `_create_backfill()` blocks new backfills by counting `Backfill.completed_at IS NULL` (`airflow/models/backfill.py:577-590`), this can leave a DAG permanently unable to start new backfills without manual DB cleanup. Consider adding a bounded “initializing” window (e.g., only require the association for very recent backfills) or introducing an explicit backfill state/failed marker so initialization failures don’t create stuck active backfills.
   ```suggestion
           # Treat very recent backfills with no associations as "initializing",
           # but allow older ones without BackfillDagRun rows to be auto-completed
           # so they don't block new backfills if initialization failed.
           initializing_cutoff = now - timedelta(minutes=5)
           # todo: AIP-78 simplify this function to an update statement
           query = select(Backfill).where(
               Backfill.completed_at.is_(None),
               or_(
                   # Backfill has at least one association and is fully initialized.
                   exists(select(BackfillDagRun.id).where(BackfillDagRun.backfill_id == Backfill.id)),
                   # Or it is older than the initializing window; treat it as no longer initializing
                   # even if it has no BackfillDagRun rows (e.g. initialization crashed).
                   Backfill.created_at < initializing_cutoff,
               ),
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org