You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vatsrahul1001 (via GitHub)" <gi...@apache.org> on 2025/01/24 16:40:10 UTC
[PR] Handle future dates cases in Backfill [airflow]
vatsrahul1001 opened a new pull request, #46017:
URL: https://github.com/apache/airflow/pull/46017
Closes: https://github.com/apache/airflow/issues/46012
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle future dates cases in Backfill [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish merged PR #46017:
URL: https://github.com/apache/airflow/pull/46017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle future dates cases in Backfill [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46017:
URL: https://github.com/apache/airflow/pull/46017#discussion_r1949669565
##########
airflow/models/backfill.py:
##########
@@ -347,7 +358,7 @@ def _get_info_list(
now = timezone.utcnow()
dagrun_info_list = (x for x in infos if x.data_interval.end < now)
if reverse:
- dagrun_info_list = reversed([x for x in dag.iter_dagrun_infos_between(from_date, to_date)])
+ dagrun_info_list = reversed([x for x in infos if x.data_interval.end < now])
Review Comment:
```suggestion
dagrun_info_list = reversed(dag_run_info_list)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle future dates cases in Backfill [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #46017:
URL: https://github.com/apache/airflow/pull/46017#discussion_r1941432267
##########
airflow/models/backfill.py:
##########
@@ -215,6 +223,8 @@ def _validate_backfill_params(dag, reverse, reprocess_behavior: ReprocessBehavio
"DAG has tasks for which depends_on_past=True. "
"You must set reprocess behavior to reprocess completed or reprocess failed."
)
+ if from_date >= timezone.utcnow() and to_date >= timezone.utcnow():
Review Comment:
timezone.utcnow() can be extracted to the same constant value. instead of called twice.
##########
airflow/models/backfill.py:
##########
@@ -215,6 +223,8 @@ def _validate_backfill_params(dag, reverse, reprocess_behavior: ReprocessBehavio
"DAG has tasks for which depends_on_past=True. "
"You must set reprocess behavior to reprocess completed or reprocess failed."
)
+ if from_date >= timezone.utcnow() and to_date >= timezone.utcnow():
Review Comment:
What if only one of the two dates (from_date or to_date) is in the future ? Should that be considered invalid or valid ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle future dates cases in Backfill [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46017:
URL: https://github.com/apache/airflow/pull/46017#discussion_r1949669565
##########
airflow/models/backfill.py:
##########
@@ -347,7 +358,7 @@ def _get_info_list(
now = timezone.utcnow()
dagrun_info_list = (x for x in infos if x.data_interval.end < now)
if reverse:
- dagrun_info_list = reversed([x for x in dag.iter_dagrun_infos_between(from_date, to_date)])
+ dagrun_info_list = reversed([x for x in infos if x.data_interval.end < now])
Review Comment:
```suggestion
dagrun_info_list = reversed(dag_run_info_list)
```
The < now filter was laready applied 2 lines above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle future dates cases in Backfill [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on code in PR #46017:
URL: https://github.com/apache/airflow/pull/46017#discussion_r1943348920
##########
airflow/models/backfill.py:
##########
@@ -215,6 +223,8 @@ def _validate_backfill_params(dag, reverse, reprocess_behavior: ReprocessBehavio
"DAG has tasks for which depends_on_past=True. "
"You must set reprocess behavior to reprocess completed or reprocess failed."
)
+ if from_date >= timezone.utcnow() and to_date >= timezone.utcnow():
Review Comment:
I think to_date is already being handled [here](https://github.com/apache/airflow/pull/46017/files#diff-f964a968e76ed62e5a64804b6c2387c790ec8ddc8aeeaced41cdc4b43413db06R338)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle future dates cases in Backfill [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #46017:
URL: https://github.com/apache/airflow/pull/46017#issuecomment-2651131504
This PR fixes the below behaviors
Ensure dates which are in future with run_backwards=true are not inserted in backfill_dag_run table
Raise 422 when all dates are in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org