You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vatsrahul1001 (via GitHub)" <gi...@apache.org> on 2025/02/07 13:40:11 UTC
[PR] AIP-83 refactor backfill reprocess behaviour [airflow]
vatsrahul1001 opened a new pull request, #46554:
URL: https://github.com/apache/airflow/pull/46554
closes: https://github.com/apache/airflow/issues/46278
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952649036
##########
tests/models/test_backfill.py:
##########
@@ -210,12 +209,8 @@ def test_reprocess_behavior(reprocess_behavior, run_counts, dag_maker, session):
# whether a dag run is created for backfill depends on
# the last run for a logical date
("2021-01-02", ["failed"]),
- ("2021-01-03", ["success", "failed"]), # <-- 2021-01-03 is "failed"
- ("2021-01-04", ["failed", "success"]), # <-- 2021-01-04 is "success"
- ("2021-01-05", ["success", "success"]),
- ("2021-01-06", ["failed", "failed"]),
- ("2021-01-07", ["running", "running"]),
- ("2021-01-08", ["failed", "running"]),
Review Comment:
instead of removing these test cases we should document what changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #46554:
URL: https://github.com/apache/airflow/pull/46554#issuecomment-2677005742
obsoleted by https://github.com/apache/airflow/pull/46959
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish closed pull request #46554: AIP-83 refactor backfill reprocess behaviour
URL: https://github.com/apache/airflow/pull/46554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952700496
##########
airflow/models/backfill.py:
##########
@@ -432,3 +435,31 @@ def _create_backfill(
info,
)
return br
+
+
+def clear_run_if_dagrun_exists(dag, info, reprocess_behavior, backfill_id, session):
+ from airflow.models import DagRun
+
+ dr = session.scalar(
+ statement=_get_latest_dag_run_row_query(info, session),
+ )
+ if dr:
+ if dr.state in {DagRunState.SUCCESS, DagRunState.FAILED} and reprocess_behavior in {
+ ReprocessBehavior.COMPLETED,
+ ReprocessBehavior.FAILED,
+ }:
+ dag.clear(
+ run_id=dr.run_id,
+ dag_run_state=DagRunState.QUEUED,
+ session=session,
+ confirm_prompt=False,
+ dry_run=False,
+ )
+ # updating backfill id and run_type in dag_run table
+ update_backfill_id = (
+ update(DagRun)
+ .where(DagRun.logical_date == info.logical_date)
+ .values(backfill_id=backfill_id, run_type=DagRunType.BACKFILL_JOB)
+ )
+ session.execute(update_backfill_id)
Review Comment:
and the reason why is, if it is part of backfill, the scheduler looks at it differently w.r.t. max active runs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952649435
##########
tests/models/test_backfill.py:
##########
@@ -271,12 +266,12 @@ def _get_bdr(date):
# 2021-01-04 is "failed" so it may or may not be reprocessed depending
# on the configuration
- bdr = _get_bdr("2021-01-04")
Review Comment:
changed the date?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952682256
##########
tests/models/test_backfill.py:
##########
@@ -271,12 +266,12 @@ def _get_bdr(date):
# 2021-01-04 is "failed" so it may or may not be reprocessed depending
# on the configuration
- bdr = _get_bdr("2021-01-04")
+ bdr = _get_bdr("2021-01-05")
actual_reason = bdr.exception_reason
if reprocess_behavior is ReprocessBehavior.FAILED:
- assert actual_reason == BackfillDagRunExceptionReason.ALREADY_EXISTS
+ assert actual_reason == BackfillDagRunExceptionReason.IN_FLIGHT
elif reprocess_behavior is ReprocessBehavior.COMPLETED:
- assert actual_reason is None
+ assert actual_reason == BackfillDagRunExceptionReason.IN_FLIGHT
Review Comment:
we sholud not add an exception reason if we're clearing an existing run i think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952670126
##########
tests/models/test_backfill.py:
##########
@@ -271,12 +266,12 @@ def _get_bdr(date):
# 2021-01-04 is "failed" so it may or may not be reprocessed depending
# on the configuration
- bdr = _get_bdr("2021-01-04")
Review Comment:
instead of changing the date, this should probably assert that exception reason is None, and assert that the dag run assorciated with this bdr is the same dag run that was created before the backfill. to document the behavior that we are reusing dag runs and we don't regard it as any kind of exception in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952703142
##########
airflow/models/backfill.py:
##########
@@ -432,3 +435,31 @@ def _create_backfill(
info,
)
return br
+
+
+def clear_run_if_dagrun_exists(dag, info, reprocess_behavior, backfill_id, session):
+ from airflow.models import DagRun
+
+ dr = session.scalar(
+ statement=_get_latest_dag_run_row_query(info, session),
+ )
+ if dr:
+ if dr.state in {DagRunState.SUCCESS, DagRunState.FAILED} and reprocess_behavior in {
+ ReprocessBehavior.COMPLETED,
+ ReprocessBehavior.FAILED,
+ }:
+ dag.clear(
+ run_id=dr.run_id,
+ dag_run_state=DagRunState.QUEUED,
+ session=session,
+ confirm_prompt=False,
+ dry_run=False,
+ )
+ # updating backfill id and run_type in dag_run table
+ update_backfill_id = (
+ update(DagRun)
+ .where(DagRun.logical_date == info.logical_date)
+ .values(backfill_id=backfill_id, run_type=DagRunType.BACKFILL_JOB)
+ )
+ session.execute(update_backfill_id)
Review Comment:
if you clear, then immediately it will be scheduled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952873404
##########
airflow/models/backfill.py:
##########
@@ -432,3 +435,31 @@ def _create_backfill(
info,
)
return br
+
+
+def clear_run_if_dagrun_exists(dag, info, reprocess_behavior, backfill_id, session):
Review Comment:
@dstandish, as soon as we clear a run, should_create_backfill_dag_run sets it to the `In Flight` state because the run remains in the queued state after clearing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952741267
##########
airflow/models/backfill.py:
##########
@@ -288,6 +289,8 @@ def _create_backfill_dag_run(
):
from airflow.models.dagrun import DagRun
+ # clear dag run if run exits and reprocess behaviour is in completed or failed
+ clear_run_if_dagrun_exists(dag, info, reprocess_behavior, backfill_id, session)
Review Comment:
i feel like `clear_run_if_dagrun_exists` should be moved into `should_create_backfill_dag_run`. we should probably only evaluate the reprocess behavior in _one place_. and it should only be evaluated against runs that are there before the backfill request, i.e. not those modified (i.e. cleared) in the backfill request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952707379
##########
airflow/models/backfill.py:
##########
@@ -432,3 +435,31 @@ def _create_backfill(
info,
)
return br
+
+
+def clear_run_if_dagrun_exists(dag, info, reprocess_behavior, backfill_id, session):
Review Comment:
we also should make sure (and make clear) in our test cases that when we clear a run for backfill, that we don't create an "exception reason" because there is no exception in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-83 refactor backfill reprocess behaviour [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46554:
URL: https://github.com/apache/airflow/pull/46554#discussion_r1952684064
##########
airflow/models/backfill.py:
##########
@@ -432,3 +435,31 @@ def _create_backfill(
info,
)
return br
+
+
+def clear_run_if_dagrun_exists(dag, info, reprocess_behavior, backfill_id, session):
+ from airflow.models import DagRun
+
+ dr = session.scalar(
+ statement=_get_latest_dag_run_row_query(info, session),
+ )
+ if dr:
+ if dr.state in {DagRunState.SUCCESS, DagRunState.FAILED} and reprocess_behavior in {
+ ReprocessBehavior.COMPLETED,
+ ReprocessBehavior.FAILED,
+ }:
+ dag.clear(
+ run_id=dr.run_id,
+ dag_run_state=DagRunState.QUEUED,
+ session=session,
+ confirm_prompt=False,
+ dry_run=False,
+ )
+ # updating backfill id and run_type in dag_run table
+ update_backfill_id = (
+ update(DagRun)
+ .where(DagRun.logical_date == info.logical_date)
+ .values(backfill_id=backfill_id, run_type=DagRunType.BACKFILL_JOB)
+ )
+ session.execute(update_backfill_id)
Review Comment:
the clear and the update should happen in the same transaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org