You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "subhash-0000 (via GitHub)" <gi...@apache.org> on 2026/01/20 10:11:56 UTC
[PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
subhash-0000 opened a new pull request, #60818:
URL: https://github.com/apache/airflow/pull/60818
## Description
This commit implements a new feature that allows users to run backfills without unpausing the DAG, addressing issue #60768.
## Problem Statement
Currently, to perform a backfill, a DAG must be unpaused. However, for DAGs that reprocess historical data but shouldn't run on their regular schedule (e.g., when daily data is not available), this creates unnecessary executions. Users need a way to run backfills while keeping the DAG paused to avoid useless computation and logs.
## Solution
Added a new "Run backfill while keeping DAG paused" option that allows backfills to execute even when the DAG is paused, without unpausing it. When enabled, only the backfill runs execute—the DAG remains paused and won't be triggered by its regular schedule.
## Changes
### Backend
- Added `keep_dag_paused` boolean column to `Backfill` model
- Modified scheduler logic in `get_queued_dag_runs_to_set_running()` and `get_running_dag_runs_to_examine()` to allow backfill execution on paused DAGs when `keep_dag_paused=True`
- Updated `BackfillPostBody` and `BackfillResponse` datamodels to include the new field
- Modified `_create_backfill()` function to accept `keep_dag_paused` parameter
- Updated `create_backfill` API route to pass `keep_dag_paused` parameter
### Frontend
- Added checkbox in `RunBackfillForm`: "Run backfill while keeping {dag_name} paused"
- Added translation string for the new checkbox option
- Checkbox is disabled when "Unpause DAG on trigger" is selected (mutually exclusive)
- Smart logic ensures `keep_dag_paused` is only sent when DAG is paused and not being unpaused
### Database
- Created Alembic migration `0100_3_2_0_add_keep_dag_paused_to_backfill.py` to add new column
### Tests
- Added unit tests for `Backfill` model with `keep_dag_paused` functionality
- Added API tests for backfill creation with `keep_dag_paused` parameter
- Added scheduler tests to verify backfills run on paused DAGs when enabled
- Updated existing tests to include `keep_dag_paused` in response assertions
### Documentation
- Added "Running backfills on paused DAGs" section to backfill documentation
- Updated UI instructions to explain both unpause options
- Documented use cases and when to use the feature
- Created newsfragment for release notes
## Backward Compatibility
✅ Fully backward compatible - `keep_dag_paused` defaults to `False`, preserving existing behavior.
## Testing
All code changes are covered with tests:
- Model tests verify backfill creation and default values
- Scheduler tests verify paused DAGs can execute backfills
- API tests verify endpoint accepts and returns the parameter correctly
Closes: #60768
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: GitHub Copilot following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2748892945
##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -598,9 +598,11 @@ def get_running_dag_runs_to_examine(cls, session: Session) -> ScalarResult[DagRu
.where(cls.state == DagRunState.RUNNING)
.join(DagModel, DagModel.dag_id == cls.dag_id)
.join(BackfillDagRun, BackfillDagRun.dag_run_id == DagRun.id, isouter=True)
+ .join(Backfill, isouter=True)
.where(
- DagModel.is_paused == false(),
DagModel.is_stale == false(),
+ # allow backfills to run on paused DAGs
+ or_(DagModel.is_paused == false(), DagRun.backfill_id.isnot(None)),
Review Comment:
The `DagRun.backfill_id.isnot(None)` condition is used to allow backfill runs to execute even when the DAG is paused. This way, regular scheduled runs are blocked if the DAG is paused, but backfills (which have a non-None `backfill_id`) can still run. This matches the requested behavior to support backfills on paused DAGs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2829116260
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst:
##########
@@ -68,9 +68,11 @@ Follow the steps below to enable custom logging config class:
.. code-block:: python
from copy import deepcopy
- from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
- LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
+ # Airflow 3.x and later uses structlog for logging configuration.
+ # The legacy DEFAULT_LOGGING_CONFIG dict is no longer respected.
+ # See the migration note below for details.
+ # Example: To customize logging, use structlog configuration as described in the Airflow documentation.
Review Comment:
Doesn't seem related to the PR
##########
airflow-core/docs/core-concepts/backfill.rst:
##########
@@ -47,6 +47,24 @@ Run ordering
You can run your backfill in reverse, i.e. latest runs first. The CLI option is ``--run-backwards``.
+Running backfills on paused DAGs
+---------------------------------
+
+Airflow allows backfills to run on paused DAGs. This is useful when:
+
+* You need to reprocess historical data without activating the regular DAG schedule
+* The data source is not available for regular runs
+* You want to avoid unnecessary computation and logs from scheduled triggers
+
+When you create a backfill on a paused DAG, only the backfill runs will execute—the DAG remains paused and will not be triggered by its regular schedule.
+
+**Via UI**: When creating a backfill for a paused DAG, you can:
+
+- **Check "Unpause on trigger"**: This will unpause the DAG before running the backfill (and regular schedules will resume)
+- **Uncheck "Unpause on trigger"** (default): The backfill will run but the DAG stays paused
Review Comment:
Did you manually test this, are the backfill runs succeeding now ?
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst:
##########
@@ -117,25 +119,11 @@ Example of custom logging for the ``SQLExecuteQueryOperator`` and the ``HttpHook
from copy import deepcopy
from pydantic.utils import deep_update
- from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
- LOGGING_CONFIG = deep_update(
- deepcopy(DEFAULT_LOGGING_CONFIG),
- {
- "loggers": {
- "airflow.task.operators.airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator": {
- "handlers": ["task"],
- "level": "DEBUG",
- "propagate": True,
- },
- "airflow.task.hooks.airflow.providers.http.hooks.http.HttpHook": {
- "handlers": ["task"],
- "level": "WARNING",
- "propagate": False,
- },
- }
- },
- )
+ # Airflow 3.x and later uses structlog for logging configuration.
+ # The legacy DEFAULT_LOGGING_CONFIG dict is no longer respected.
+ # See the migration note below for details.
+ # Example: To customize logging, use structlog configuration as described in the Airflow documentation.
Review Comment:
Same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on PR #60818:
URL: https://github.com/apache/airflow/pull/60818#issuecomment-3857896408
Hi @potiuk and @jroachgolf84, just a final nudge to get this unblocked.
I have refactored the implementation to use existing logic as requested. I used DagRun.backfill_id.isnot(None) specifically as a discriminator: it allows the scheduler to identify and run user-initiated backfills even when the DAG is paused, while still blocking regular scheduled runs.
All CI checks are green and the code is simplified. Could you please take a final look? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2777171784
##########
airflow-core/docs/core-concepts/backfill.rst:
##########
@@ -47,6 +47,24 @@ Run ordering
You can run your backfill in reverse, i.e. latest runs first. The CLI option is ``--run-backwards``.
+Running backfills on paused DAGs
+---------------------------------
+
+Airflow allows backfills to run on paused DAGs. This is useful when:
+
+* You need to reprocess historical data without activating the regular DAG schedule
+* The data source is not available for regular runs
+* You want to avoid unnecessary computation and logs from scheduled triggers
+
+When you create a backfill on a paused DAG, only the backfill runs will execute—the DAG remains paused and will not be triggered by its regular schedule.
+
+**Via UI**: When creating a backfill for a paused DAG, you can:
+
+- **Check "Unpause on trigger"**: This will unpause the DAG before running the backfill (and regular schedules will resume)
+- **Uncheck "Unpause on trigger"** (default): The backfill will run but the DAG stays paused
Review Comment:
Hi @pierrejeambrun, you're absolutely right—I apologize for not testing this manually earlier. My unit tests passed, but they didn't capture the full scheduler integration.
Root Cause:
While the DagRuns were created, the Task Instances were being blocked at three checkpoints in the scheduler:
Scheduler Task Query: The query wasn't fetching TIs from paused DAGs even if they were part of a backfill.
Task Dependency Check: Tasks were failing the DagUnpausedDep validation.
State Update: The scheduler was actively trying to set backfill runs to a finished state because the DAG was paused.
Fixes Applied:
I've added backfill_id exceptions to:
scheduler_job_runner.py (Task query logic and state update loop)
dag_unpaused_dep.py (Bypassing the unpaused dependency for backfills)
The changes are pushed and CI is running. Could you please take another look? The tasks should now transition from Queued to Running as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2831764211
##########
airflow-core/docs/core-concepts/backfill.rst:
##########
@@ -47,6 +47,24 @@ Run ordering
You can run your backfill in reverse, i.e. latest runs first. The CLI option is ``--run-backwards``.
+Running backfills on paused DAGs
+---------------------------------
+
+Airflow allows backfills to run on paused DAGs. This is useful when:
+
+* You need to reprocess historical data without activating the regular DAG schedule
+* The data source is not available for regular runs
+* You want to avoid unnecessary computation and logs from scheduled triggers
+
+When you create a backfill on a paused DAG, only the backfill runs will execute—the DAG remains paused and will not be triggered by its regular schedule.
+
+**Via UI**: When creating a backfill for a paused DAG, you can:
+
+- **Check "Unpause on trigger"**: This will unpause the DAG before running the backfill (and regular schedules will resume)
+- **Uncheck "Unpause on trigger"** (default): The backfill will run but the DAG stays paused
Review Comment:
Manually tested with Breeze (Airflow 3.2.0.dev0). Screenshot shows:
DAG toggle is off/paused throughout
Two backfill runs created with Run Type = Backfill
Both runs have actual Start Dates and durations — tasks executed and were not stuck in Queued as in the previous iteration
The Failed state is due to example_bash_operator's bash commands failing in the Breeze container environment, unrelated to the paused DAG behavior.
<img width="1401" height="742" alt="image" src="https://github.com/user-attachments/assets/9cbbe0e7-30a7-4450-896e-9b6524545b90" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2831783738
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst:
##########
@@ -117,25 +119,11 @@ Example of custom logging for the ``SQLExecuteQueryOperator`` and the ``HttpHook
from copy import deepcopy
from pydantic.utils import deep_update
- from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
- LOGGING_CONFIG = deep_update(
- deepcopy(DEFAULT_LOGGING_CONFIG),
- {
- "loggers": {
- "airflow.task.operators.airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator": {
- "handlers": ["task"],
- "level": "DEBUG",
- "propagate": True,
- },
- "airflow.task.hooks.airflow.providers.http.hooks.http.HttpHook": {
- "handlers": ["task"],
- "level": "WARNING",
- "propagate": False,
- },
- }
- },
- )
+ # Airflow 3.x and later uses structlog for logging configuration.
+ # The legacy DEFAULT_LOGGING_CONFIG dict is no longer respected.
+ # See the migration note below for details.
+ # Example: To customize logging, use structlog configuration as described in the Airflow documentation.
Review Comment:
Covered by the same revert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2748892945
##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -598,9 +598,11 @@ def get_running_dag_runs_to_examine(cls, session: Session) -> ScalarResult[DagRu
.where(cls.state == DagRunState.RUNNING)
.join(DagModel, DagModel.dag_id == cls.dag_id)
.join(BackfillDagRun, BackfillDagRun.dag_run_id == DagRun.id, isouter=True)
+ .join(Backfill, isouter=True)
.where(
- DagModel.is_paused == false(),
DagModel.is_stale == false(),
+ # allow backfills to run on paused DAGs
+ or_(DagModel.is_paused == false(), DagRun.backfill_id.isnot(None)),
Review Comment:
The `DagRun.backfill_id.isnot(None)` condition is used to allow backfill runs to execute even when the DAG is paused. This way, regular scheduled runs are blocked if the DAG is paused, but backfills (which have a non-None `backfill_id`) can still run. This matches the requested behavior to support backfills on paused DAGs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2774133593
##########
airflow-core/docs/core-concepts/backfill.rst:
##########
@@ -47,6 +47,24 @@ Run ordering
You can run your backfill in reverse, i.e. latest runs first. The CLI option is ``--run-backwards``.
+Running backfills on paused DAGs
+---------------------------------
+
+Airflow allows backfills to run on paused DAGs. This is useful when:
+
+* You need to reprocess historical data without activating the regular DAG schedule
+* The data source is not available for regular runs
+* You want to avoid unnecessary computation and logs from scheduled triggers
+
+When you create a backfill on a paused DAG, only the backfill runs will execute—the DAG remains paused and will not be triggered by its regular schedule.
+
+**Via UI**: When creating a backfill for a paused DAG, you can:
+
+- **Check "Unpause on trigger"**: This will unpause the DAG before running the backfill (and regular schedules will resume)
+- **Uncheck "Unpause on trigger"** (default): The backfill will run but the DAG stays paused
Review Comment:
Did you actually test this ? Dags aren't scheduled.
<img width="1919" height="970" alt="Image" src="https://github.com/user-attachments/assets/d489a74a-8452-452d-8618-1bebb7ce709c" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on PR #60818:
URL: https://github.com/apache/airflow/pull/60818#issuecomment-3822111316
Hi @jroachgolf84,
just following up—this is ready for re-review whenever you have a chance. All CI checks are passing and the implementation now uses the existing unpause logic as requested. Please let me know if you need anything else
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun closed pull request #60818: Add support for running backfill while keeping DAG paused (fixes #60768)
URL: https://github.com/apache/airflow/pull/60818
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #60818:
URL: https://github.com/apache/airflow/pull/60818#issuecomment-3774288159
I'm not sure how much I love naming it `keep_dag_paused`. Have you taken a peek at the "unpause" logic that is currently in-place w.r.t. backfills? I think it makes sense to take a look at that and work with that functionality rather than making a (pretty significant) change like this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on PR #60818:
URL: https://github.com/apache/airflow/pull/60818#issuecomment-3796641382
@jroachgolf84
] I've addressed your feedback. Removed the `keep_dag_paused` field entirely and simplified the implementation to work with the existing unpause functionality. The scheduler now allows backfills on paused DAGs by checking for `backfill_id`, and the UI uses the existing unpause checkbox.
All CI checks are now passing. Ready for re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2831782258
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst:
##########
@@ -68,9 +68,11 @@ Follow the steps below to enable custom logging config class:
.. code-block:: python
from copy import deepcopy
- from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
- LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
+ # Airflow 3.x and later uses structlog for logging configuration.
+ # The legacy DEFAULT_LOGGING_CONFIG dict is no longer respected.
+ # See the migration note below for details.
+ # Example: To customize logging, use structlog configuration as described in the Airflow documentation.
Review Comment:
Reverted entire advanced-logging-configuration.rst to origin/main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2812262744
##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -598,9 +598,11 @@ def get_running_dag_runs_to_examine(cls, session: Session) -> ScalarResult[DagRu
.where(cls.state == DagRunState.RUNNING)
.join(DagModel, DagModel.dag_id == cls.dag_id)
.join(BackfillDagRun, BackfillDagRun.dag_run_id == DagRun.id, isouter=True)
+ .join(Backfill, isouter=True)
.where(
- DagModel.is_paused == false(),
DagModel.is_stale == false(),
+ # allow backfills to run on paused DAGs
+ or_(DagModel.is_paused == false(), DagRun.backfill_id.isnot(None)),
Review Comment:
Makes sense, thanks for the explanation!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2746616915
##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -598,9 +598,11 @@ def get_running_dag_runs_to_examine(cls, session: Session) -> ScalarResult[DagRu
.where(cls.state == DagRunState.RUNNING)
.join(DagModel, DagModel.dag_id == cls.dag_id)
.join(BackfillDagRun, BackfillDagRun.dag_run_id == DagRun.id, isouter=True)
+ .join(Backfill, isouter=True)
.where(
- DagModel.is_paused == false(),
DagModel.is_stale == false(),
+ # allow backfills to run on paused DAGs
+ or_(DagModel.is_paused == false(), DagRun.backfill_id.isnot(None)),
Review Comment:
Why are you using the `DagRun.backfill_id.isnot(None)`? Just curious the logic here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2749142085
##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -598,9 +598,11 @@ def get_running_dag_runs_to_examine(cls, session: Session) -> ScalarResult[DagRu
.where(cls.state == DagRunState.RUNNING)
.join(DagModel, DagModel.dag_id == cls.dag_id)
.join(BackfillDagRun, BackfillDagRun.dag_run_id == DagRun.id, isouter=True)
+ .join(Backfill, isouter=True)
.where(
- DagModel.is_paused == false(),
DagModel.is_stale == false(),
+ # allow backfills to run on paused DAGs
+ or_(DagModel.is_paused == false(), DagRun.backfill_id.isnot(None)),
Review Comment:
The reason for using `DagRun.backfill_id.isnot(None)` is to allow backfill runs to execute even when the DAG is paused. This way, regular scheduled runs are blocked if the DAG is paused, but backfills (which have a non-None `backfill_id`) can still run. This directly addresses the feature request to support backfills on paused DAGs without triggering regular daily executions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for running backfill while keeping DAG paused (fixes #60768) [airflow]
Posted by "subhash-0000 (via GitHub)" <gi...@apache.org>.
subhash-0000 commented on code in PR #60818:
URL: https://github.com/apache/airflow/pull/60818#discussion_r2757096473
##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -598,9 +598,11 @@ def get_running_dag_runs_to_examine(cls, session: Session) -> ScalarResult[DagRu
.where(cls.state == DagRunState.RUNNING)
.join(DagModel, DagModel.dag_id == cls.dag_id)
.join(BackfillDagRun, BackfillDagRun.dag_run_id == DagRun.id, isouter=True)
+ .join(Backfill, isouter=True)
.where(
- DagModel.is_paused == false(),
DagModel.is_stale == false(),
+ # allow backfills to run on paused DAGs
+ or_(DagModel.is_paused == false(), DagRun.backfill_id.isnot(None)),
Review Comment:
@jroachgolf84 Does my explanation address your question? Let me know if you'd like me to clarify further or make any changes. Happy to adjust the implementation if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org