You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ashb (via GitHub)" <gi...@apache.org> on 2025/04/22 13:43:13 UTC
[PR] Make LatestOnlyOperator work for default data-interval-less DAGs [airflow]
ashb opened a new pull request, #49554:
URL: https://github.com/apache/airflow/pull/49554
Remove the check/don't skip logic whe the data interval is zero-wdith. Even if
a DAG doesn't have the concept of a data-interval (i.e. it is zero width) it
still is logically consistent for it to have to concept of latest or not, so
we now only compare against the end date of the interval.
(And a few drive-by refactors too, `context["task"]` is `self`,
`context["dag"]` is `self.dag`)
<img width="391" alt="Screenshot 2025-04-22 at 14 20 05" src="https://github.com/user-attachments/assets/cdba2a43-2a61-4556-b2ef-e2fcf33738a7" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make LatestOnlyOperator work for default data-interval-less DAGs [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #49554:
URL: https://github.com/apache/airflow/pull/49554#discussion_r2054157789
##########
providers/standard/tests/unit/standard/operators/test_latest_only_operator.py:
##########
@@ -164,78 +164,47 @@ def test_skipping_non_latest(self, dag_maker):
latest_ti2 = dr2.get_task_instance(task_id="latest")
latest_ti2.task = latest_task
latest_ti2.run()
-
- latest_ti2.set_state(State.SUCCESS)
-
- # Verify the state of the other downstream tasks
- downstream_task.run(start_date=DEFAULT_DATE, end_date=END_DATE)
- downstream_task2.run(start_date=DEFAULT_DATE, end_date=END_DATE)
- downstream_task3.run(start_date=DEFAULT_DATE, end_date=END_DATE)
-
- downstream_instances = get_task_instances("downstream")
- exec_date_to_downstream_state = {ti.logical_date: ti.state for ti in downstream_instances}
- assert exec_date_to_downstream_state == {
- timezone.datetime(2016, 1, 1): "skipped",
- timezone.datetime(2016, 1, 1, 12): "skipped",
- timezone.datetime(2016, 1, 2): "success",
- }
-
- downstream_instances = get_task_instances("downstream_2")
- exec_date_to_downstream_state = {ti.logical_date: ti.state for ti in downstream_instances}
- assert exec_date_to_downstream_state == {
- timezone.datetime(2016, 1, 1): None,
- timezone.datetime(2016, 1, 1, 12): None,
- timezone.datetime(2016, 1, 2): "success",
- }
-
- downstream_instances = get_task_instances("downstream_3")
- exec_date_to_downstream_state = {ti.logical_date: ti.state for ti in downstream_instances}
- assert exec_date_to_downstream_state == {
- timezone.datetime(2016, 1, 1): "success",
- timezone.datetime(2016, 1, 1, 12): "success",
- timezone.datetime(2016, 1, 2): "success",
- }
-
else:
latest_task.run(start_date=DEFAULT_DATE, end_date=END_DATE)
- downstream_task.run(start_date=DEFAULT_DATE, end_date=END_DATE)
- downstream_task2.run(start_date=DEFAULT_DATE, end_date=END_DATE)
- downstream_task3.run(start_date=DEFAULT_DATE, end_date=END_DATE)
+ latest_instances = get_task_instances("latest")
+ exec_date_to_latest_state = {ti.logical_date: ti.state for ti in latest_instances}
+ assert exec_date_to_latest_state == {
+ timezone.datetime(2016, 1, 1): "success",
+ timezone.datetime(2016, 1, 1, 12): "success",
+ timezone.datetime(2016, 1, 2): "success",
+ }
- latest_instances = get_task_instances("latest")
Review Comment:
All of these changes in the tests are to reduce the differences between v2 and v3 -- it should behave the same.
Best viewed in "split mode"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make LatestOnlyOperator work for default data-interval-less DAGs [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #49554:
URL: https://github.com/apache/airflow/pull/49554#discussion_r2056368810
##########
providers/standard/src/airflow/providers/standard/operators/latest_only.py:
##########
@@ -29,8 +30,8 @@
from airflow.utils.types import DagRunType
if TYPE_CHECKING:
- from airflow.models import DAG, DagRun
- from airflow.timetables.base import DagRunInfo
+ from airflow.models import DagRun
+ from pendulum.datetime import DateTime
Review Comment:
```suggestion
from pendulum.datetime import DateTime
from airflow.models import DagRun
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make LatestOnlyOperator work for default data-interval-less DAGs [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #49554:
URL: https://github.com/apache/airflow/pull/49554#discussion_r2054789156
##########
providers/standard/src/airflow/providers/standard/operators/latest_only.py:
##########
@@ -62,54 +63,65 @@ def choose_branch(self, context: Context) -> str | Iterable[str]:
dag_run: DagRun = context["dag_run"] # type: ignore[assignment]
if dag_run.run_type == DagRunType.MANUAL:
self.log.info("Manually triggered DAG_Run: allowing execution to proceed.")
- return list(context["task"].get_direct_relative_ids(upstream=False))
+ return list(self.get_direct_relative_ids(upstream=False))
- next_info = self._get_next_run_info(context, dag_run)
- now = pendulum.now("UTC")
+ dates = self._get_compare_dates(dag_run)
- if next_info is None:
+ if dates is None:
self.log.info("Last scheduled execution: allowing execution to proceed.")
- return list(context["task"].get_direct_relative_ids(upstream=False))
+ return list(self.get_direct_relative_ids(upstream=False))
- left_window, right_window = next_info.data_interval
+ now = pendulum.now("UTC")
+ left_window, right_window = dates
self.log.info(
"Checking latest only with left_window: %s right_window: %s now: %s",
left_window,
right_window,
now,
)
- if left_window == right_window:
- self.log.info(
- "Zero-length interval [%s, %s) from timetable (%s); treating current run as latest.",
- left_window,
- right_window,
- self.dag.timetable.__class__,
- )
- return list(context["task"].get_direct_relative_ids(upstream=False))
-
if not left_window < now <= right_window:
self.log.info("Not latest execution, skipping downstream.")
# we return an empty list, thus the parent BaseBranchOperator
# won't exclude any downstream tasks from skipping.
return []
- self.log.info("Latest, allowing execution to proceed.")
- return list(context["task"].get_direct_relative_ids(upstream=False))
- def _get_next_run_info(self, context: Context, dag_run: DagRun) -> DagRunInfo | None:
- dag: DAG = context["dag"] # type: ignore[assignment]
+ self.log.info("Latest, allowing execution to proceed.")
+ return list(self.get_direct_relative_ids(upstream=False))
+ def _get_compare_dates(self, dag_run: DagRun) -> tuple[DateTime, DateTime] | None:
+ dagrun_date: DateTime
if AIRFLOW_V_3_0_PLUS:
- from airflow.timetables.base import DataInterval, TimeRestriction
+ dagrun_date = dag_run.logical_date or dag_run.run_after
+ else:
+ dagrun_date = dag_run.logical_date
- time_restriction = TimeRestriction(earliest=None, latest=None, catchup=True)
- current_interval = DataInterval(start=dag_run.data_interval_start, end=dag_run.data_interval_end)
+ # breakpoint()
Review Comment:
I assume this is a testing leftover/glitch?
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make LatestOnlyOperator work for default data-interval-less DAGs [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #49554:
URL: https://github.com/apache/airflow/pull/49554#discussion_r2055696295
##########
providers/standard/src/airflow/providers/standard/operators/latest_only.py:
##########
@@ -29,8 +30,8 @@
from airflow.utils.types import DagRunType
if TYPE_CHECKING:
- from airflow.models import DAG, DagRun
- from airflow.timetables.base import DagRunInfo
+ from airflow.models import DagRun
+ from airflow.utils.timezone import DateTime
Review Comment:
```suggestion
from pendulum.datetime import DateTime
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make LatestOnlyOperator work for default data-interval-less DAGs [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #49554:
URL: https://github.com/apache/airflow/pull/49554#discussion_r2054159972
##########
providers/standard/tests/unit/standard/operators/test_latest_only_operator.py:
##########
@@ -115,9 +115,7 @@ def test_skipping_non_latest(self, dag_maker):
start_date=timezone.utcnow(),
logical_date=timezone.datetime(2016, 1, 1, 12),
state=State.RUNNING,
- data_interval=DataInterval(
- timezone.datetime(2016, 1, 1, 12), timezone.datetime(2016, 1, 1, 12) + INTERVAL
- ),
+ data_interval=DataInterval(timezone.datetime(2016, 1, 1, 12), timezone.datetime(2016, 1, 1, 12)),
Review Comment:
This is the crux of the test change -- interval start = interval end, which is what you get by default when you create the scheduler creates DagRuns from a DAG with a `timedelta` schedule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make LatestOnlyOperator work for default data-interval-less DAGs [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb merged PR #49554:
URL: https://github.com/apache/airflow/pull/49554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org