You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "manipatnam (via GitHub)" <gi...@apache.org> on 2026/03/24 11:59:29 UTC
[PR] Enhanced run_immediately functionality [airflow]
manipatnam opened a new pull request, #64162:
URL: https://github.com/apache/airflow/pull/64162
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
Fix `CronTriggerTimetable` ignoring `run_immediately` when `start_date` is set, simplify its
semantics, and document the parameter properly.
**Bug:** `run_immediately` was only respected on the very first run when `start_date=None`.
When `start_date` was set (the common case), `_calc_first_run()` was skipped and the
timetable always fell back to `align_to_prev(now)`, behaving as `run_immediately=True`
regardless of the value passed. The fix ensures `_calc_first_run()` is always called on
the first run, whether or not `start_date` is set.
**Semantic fix:** `run_immediately=False` previously applied an undocumented auto-buffer window
(10 % of the cron interval, minimum 5 min) identical to `None`, making `False` not mean what
users expected. `False` now cleanly means "always skip the past cron point and wait for the
next future one".
**Backward compatibility:** The default is changed from `False` to `True` so that existing DAGs
without an explicit `run_immediately` continue to run the most recent past cron point immediately,
preserving the original behaviour of `CronTriggerTimetable`.
**Changes:**
- `_TriggerTimetable.next_dagrun_info`: on first run (`no prior runs`), always call
`_calc_first_run()` regardless of `start_date`; after a pause/resume keep the existing
"pick most recent past boundary" logic unchanged
- `CronTriggerTimetable._calc_first_run`: simplified to three clean cases —
`True` (run past), `False` (wait for next), `timedelta` (run past if within window)
- Default `run_immediately` changed `False → True` in `CronTriggerTimetable`,
`MultipleCronTriggerTimetable`, and `CronPartitionTimetable`
- Docs: added full `run_immediately` documentation with examples and `versionadded:: 3.0.0`
- Docs: corrected the paused-then-unpaused example (both timetables skip only Feb 1 and
immediately trigger Feb 2 — the claim that `CronTriggerTimetable` skips Feb 2 as well was wrong)
- Tests: updated existing tests and added `test_run_immediately_false_with_start_date`,
`test_run_immediately_true_with_start_date`, `test_run_immediately_false_after_unpause`
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Sonnet 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3578416205
##########
airflow-core/docs/authoring-and-scheduling/timetable.rst:
##########
@@ -151,6 +151,63 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
def example_dag():
pass
+.. versionadded:: 3.0.0
+ The ``run_immediately`` argument was introduced in Airflow 3.
Review Comment:
Removed the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3578903768
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -87,12 +87,15 @@ def next_dagrun_info(
else:
next_start_time = self._align_to_next(restriction.earliest)
else:
- start_time_candidates = [self._align_to_prev(coerce_datetime(utcnow()))]
if last_automated_data_interval is not None:
- start_time_candidates.append(self._get_next(last_automated_data_interval.end))
- elif restriction.earliest is None:
- # Run immediately has no effect if there is restriction on earliest
- start_time_candidates.append(self._calc_first_run())
+ # _calc_first_run respects run_immediately to decide between the
+ # most recent past cron point and the next future one.
Review Comment:
run_immediately only exists on the Cron side; DeltaTriggerTimetable's _calc_first_run() doesn't know anything about it. So mentioning run_immediately here was misleading for half the classes using this method — good catch.
Regarding the explanation of that statement, I have mentioned it in the next comment
https://github.com/apache/airflow/pull/64162#discussion_r3578873499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3457511042
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -186,7 +187,7 @@ def __init__(
*,
timezone: str | Timezone | FixedTimezone,
interval: datetime.timedelta | relativedelta = datetime.timedelta(),
- run_immediately: bool | datetime.timedelta = False,
+ run_immediately: bool | datetime.timedelta = True,
Review Comment:
Changing the default is a breaking change that can’t be made until Airflow 4.0. We cannot accept this change (for about the next three years at least).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3025333325
##########
airflow-core/docs/authoring-and-scheduling/timetable.rst:
##########
@@ -151,6 +151,63 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
def example_dag():
pass
+.. versionadded:: 3.0.0
+ The ``run_immediately`` argument was introduced in Airflow 3.
+
+The optional ``run_immediately`` argument controls which cron point is scheduled when a Dag is first
+enabled or re-enabled after a pause. It has no effect when ``catchup=True`` or when prior Dag runs
+already exist (in those cases the scheduler always continues from where it left off).
+
Review Comment:
The docs say `run_immediately` affects a DAG when it is “first enabled or re-enabled after a pause”, but the implementation (and the tests added in this PR) state `run_immediately` only affects the very first run; after pause/resume with prior runs, the most recent past boundary is always scheduled. Please reword this paragraph to avoid implying `run_immediately` changes pause/resume behavior.
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -399,7 +395,7 @@ def __init__(
*,
timezone: str | Timezone | FixedTimezone,
run_offset: int | datetime.timedelta | relativedelta | None = None,
- run_immediately: bool | datetime.timedelta = False,
+ run_immediately: bool | datetime.timedelta = True,
# todo: AIP-76 we can't infer partition date from this, so we need to store it separately.
key_format: str = r"%Y-%m-%dT%H:%M:%S",
Review Comment:
`CronPartitionTimetable`’s docstring description of `run_immediately` still documents the old semantics (including `None` and the auto-buffer window). Since `CronPartitionTimetable` inherits the updated `_calc_first_run()` behavior and now defaults `run_immediately=True`, the docstring should be updated to match the new three-case semantics and default.
##########
airflow-core/docs/authoring-and-scheduling/timetable.rst:
##########
@@ -151,6 +151,63 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
def example_dag():
pass
+.. versionadded:: 3.0.0
+ The ``run_immediately`` argument was introduced in Airflow 3.
+
+The optional ``run_immediately`` argument controls which cron point is scheduled when a Dag is first
+enabled or re-enabled after a pause. It has no effect when ``catchup=True`` or when prior Dag runs
+already exist (in those cases the scheduler always continues from where it left off).
+
+* ``run_immediately=True`` *(default)* — schedule the **most recent past** cron point immediately.
+* ``run_immediately=False`` — skip the past cron point and wait for the **next future** cron point.
+* ``run_immediately=timedelta(...)`` — schedule the most recent past cron point only if it fired
+ within the given window; otherwise wait for the **next future** cron point.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.timetables.trigger import CronTriggerTimetable
+
+
+ @dag(
+ # Runs every 10 minutes.
+ # run_immediately=False: on first enable, skip any past slot that is more than
+ # 5 minutes old (the minimum buffer) and wait for the next 10-minute boundary.
+ schedule=CronTriggerTimetable(
+ "*/10 * * * *",
+ timezone="UTC",
+ run_immediately=False,
+ ),
+ start_date=datetime(2024, 1, 1),
+ catchup=False,
Review Comment:
This example comment still refers to the removed “minimum buffer” behavior (“skip any past slot that is more than 5 minutes old”). With the simplified semantics, `run_immediately=False` always skips the past cron point regardless of how recently it fired. Also, the snippet uses `datetime(...)` in `start_date` but doesn’t import `datetime` (only `timedelta`).
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -186,7 +187,7 @@ def __init__(
*,
timezone: str | Timezone | FixedTimezone,
interval: datetime.timedelta | relativedelta = datetime.timedelta(),
- run_immediately: bool | datetime.timedelta = False,
+ run_immediately: bool | datetime.timedelta = True,
) -> None:
super().__init__(cron, timezone)
self._interval = interval
Review Comment:
Constructor default for `run_immediately` is now `True`, but `CronTriggerTimetable.deserialize()` still uses `data.get("run_immediately", False)` as the fallback when the key is missing. If older serialized payloads omit the field, deserialization will set `run_immediately=False` and (with the bug fixed) change scheduling behavior unexpectedly. Consider switching the fallback to `True` (or handling legacy payload versions explicitly) and adding a regression test for deserializing without the field.
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -399,7 +395,7 @@ def __init__(
*,
timezone: str | Timezone | FixedTimezone,
run_offset: int | datetime.timedelta | relativedelta | None = None,
- run_immediately: bool | datetime.timedelta = False,
+ run_immediately: bool | datetime.timedelta = True,
# todo: AIP-76 we can't infer partition date from this, so we need to store it separately.
key_format: str = r"%Y-%m-%dT%H:%M:%S",
) -> None:
Review Comment:
`CronPartitionTimetable` now defaults `run_immediately=True`, but its `deserialize()` implementation still falls back to `False` when `run_immediately` is missing (`data.get("run_immediately", False)`). For legacy serialized DAGs without the field, this can change behavior after upgrade. Align the deserialize fallback with the new default (or version-gate it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3457498364
##########
airflow-core/docs/authoring-and-scheduling/timetable.rst:
##########
@@ -151,6 +151,63 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
def example_dag():
pass
+.. versionadded:: 3.0.0
+ The ``run_immediately`` argument was introduced in Airflow 3.
+
+The optional ``run_immediately`` argument controls which cron point is scheduled when a Dag is first
+enabled or re-enabled after a pause. It has no effect when ``catchup=True`` (in that case the
+scheduler always continues from where it left off).
+
+* ``run_immediately=True`` *(default)* — schedule the **most recent past** cron point immediately.
+* ``run_immediately=False`` — skip the past cron point and wait for the **next future** cron point.
+* ``run_immediately=timedelta(...)`` — schedule the most recent past cron point only if it fired
+ within the given window; otherwise wait for the **next future** cron point.
+
+.. code-block:: python
+
+ from datetime import datetime, timedelta
+
+ from airflow.timetables.trigger import CronTriggerTimetable
+
+
+ @dag(
+ # Runs every 10 minutes.
+ # run_immediately=False: always skip the most recent past slot and wait
+ # for the next 10-minute boundary.
+ schedule=CronTriggerTimetable(
+ "*/10 * * * *",
+ timezone="UTC",
+ run_immediately=False,
+ ),
+ start_date=datetime(2024, 1, 1),
+ catchup=False,
+ ...,
+ )
+ def example_dag():
+ pass
+
+
+ @dag(
+ # Runs hourly.
+ # run_immediately=timedelta(minutes=10): run the most recent past slot
+ # only if it fired within the last 10 minutes; otherwise wait for next.
+ schedule=CronTriggerTimetable(
+ "0 * * * *",
+ timezone="UTC",
+ run_immediately=timedelta(minutes=10),
+ ),
+ start_date=datetime(2024, 1, 1),
+ catchup=False,
+ ...,
+ )
+ def example_dag_with_buffer():
+ pass
+
+.. note::
+
+ ``run_immediately`` is a parameter of ``CronTriggerTimetable``, **not** of the ``DAG``
+ constructor. Passing it directly to ``DAG(run_immediately=...)`` has no effect.
Review Comment:
It does have an effect; your dag file will error because the argument does not exist. This should say _Passing it directly … is an error._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3578873499
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -485,15 +477,15 @@ def next_dagrun_info_v2(
else:
next_start_time = self._align_to_next(restriction.earliest)
else:
- prev_candidate = self._align_to_prev(coerce_datetime(utcnow()))
- start_time_candidates = [prev_candidate]
if last_dagrun_info is not None:
- next_candidate = self._get_next(last_dagrun_info.run_after)
- start_time_candidates.append(next_candidate)
- elif restriction.earliest is None:
- # Run immediately has no effect if there is restriction on earliest
- first_run = self._calc_first_run()
- start_time_candidates.append(first_run)
+ # _calc_first_run respects run_immediately to decide between the
+ # most recent past cron point and the next future one.
+ start_time_candidates = [
+ self._calc_first_run(),
+ self._get_next(last_dagrun_info.run_after),
+ ]
+ else:
Review Comment:
This one's actually fine — CronPartitionTimetable extends CronTriggerTimetable, so run_immediately really is in play here. The reason I left the note in: without it, it's not obvious why we're calling _calc_first_run() again on a resume, instead of just jumping straight to the most recent past boundary — this line is what makes sure a paused-then-resumed Dag still honours whatever run_immediately says, rather than ignoring it once there's history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3042837660
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -186,7 +187,7 @@ def __init__(
*,
timezone: str | Timezone | FixedTimezone,
interval: datetime.timedelta | relativedelta = datetime.timedelta(),
- run_immediately: bool | datetime.timedelta = False,
+ run_immediately: bool | datetime.timedelta = True,
) -> None:
super().__init__(cron, timezone)
self._interval = interval
Review Comment:
Both CronTriggerTimetable.deserialize() and CronPartitionTimetable.deserialize() fallbacks have been updated to True to match the constructor default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-4775749530
@manipatnam needs a rebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3042841480
##########
airflow-core/docs/authoring-and-scheduling/timetable.rst:
##########
@@ -151,6 +151,63 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
def example_dag():
pass
+.. versionadded:: 3.0.0
+ The ``run_immediately`` argument was introduced in Airflow 3.
+
+The optional ``run_immediately`` argument controls which cron point is scheduled when a Dag is first
+enabled or re-enabled after a pause. It has no effect when ``catchup=True`` or when prior Dag runs
+already exist (in those cases the scheduler always continues from where it left off).
+
Review Comment:
The implementation has been updated to match the docs — the pause/resume branch now calls _calc_first_run() instead of unconditionally using _align_to_prev(now), so run_immediately is respected after pause/resume as well. With run_immediately=True (the default), behaviour is unchanged; with False or a timedelta, the scheduler will now correctly skip/gate the most recent past cron point after unpause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3410186009
##########
airflow-core/docs/authoring-and-scheduling/timetable.rst:
##########
@@ -151,6 +151,63 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
def example_dag():
pass
+.. versionadded:: 3.0.0
Review Comment:
This is not up-to-date anymore. It if slips into 3.3.0 it should be:
```suggestion
.. versionadded:: 3.3.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@airflow.apache.org>.
github-actions[bot] closed pull request #64162: Enhanced run_immediately functionality
URL: https://github.com/apache/airflow/pull/64162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3457489662
##########
airflow-core/docs/authoring-and-scheduling/timetable.rst:
##########
@@ -151,6 +151,63 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
def example_dag():
pass
+.. versionadded:: 3.0.0
+ The ``run_immediately`` argument was introduced in Airflow 3.
Review Comment:
This line is not needed. The `versionadded` directive above covers this description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3410187879
##########
airflow-core/docs/authoring-and-scheduling/timetable.rst:
##########
@@ -151,6 +151,63 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
def example_dag():
pass
+.. versionadded:: 3.0.0
Review Comment:
Oh, it was already in 3.0.0 but docs were missing, sorry, mis-interpreted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-6457265602
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-4122530747
@SameerMesiah97
That was my actual intention modified description accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "collinmcnulty (via GitHub)" <gi...@apache.org>.
collinmcnulty commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-4118386068
Can you fill in this table so the intent is clear?
| start_date | run_immediately | behavior before this PR | behavior after this PR |
| -----------|-----------------|---------------------------|-----------------------|
| set | default | |
| set | explicit True | | |
| set | explicit False | | |
| None | default | |
| None | explicit True | | |
| None | explicit False | | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3042826584
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -399,7 +395,7 @@ def __init__(
*,
timezone: str | Timezone | FixedTimezone,
run_offset: int | datetime.timedelta | relativedelta | None = None,
- run_immediately: bool | datetime.timedelta = False,
+ run_immediately: bool | datetime.timedelta = True,
# todo: AIP-76 we can't infer partition date from this, so we need to store it separately.
key_format: str = r"%Y-%m-%dT%H:%M:%S",
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-5568029713
@manipatnam can you resolve conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3457514899
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -485,15 +477,15 @@ def next_dagrun_info_v2(
else:
next_start_time = self._align_to_next(restriction.earliest)
else:
- prev_candidate = self._align_to_prev(coerce_datetime(utcnow()))
- start_time_candidates = [prev_candidate]
if last_dagrun_info is not None:
- next_candidate = self._get_next(last_dagrun_info.run_after)
- start_time_candidates.append(next_candidate)
- elif restriction.earliest is None:
- # Run immediately has no effect if there is restriction on earliest
- first_run = self._calc_first_run()
- start_time_candidates.append(first_run)
+ # _calc_first_run respects run_immediately to decide between the
+ # most recent past cron point and the next future one.
+ start_time_candidates = [
+ self._calc_first_run(),
+ self._get_next(last_dagrun_info.run_after),
+ ]
+ else:
Review Comment:
Same as mentioned above; comment is confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-4124253984
> The changes around run_immediately=False and removing the implicit buffer feel more like a semantic change than a bug fix. The buffer logic looks quite intentional and provides a small grace window for slightly late scheduling, and now False strictly skips the past run. Also changing the default from False to True is prettty big shift. Now, I will caveat this by stating that I may be missing context, so I am happy to be corrected but a semantic shift like this in scheduler-adjacent code needs stronger justification than what you have provided.
Thanks for the review @SameerMesiah97
**Why the auto-buffer was removed from `False`**
In the old code, `False` and `None` had identical implementations — both applied the same
10% grace window. So passing `False` did not guarantee skipping the past cron point; it
could still run it if the scheduler evaluated within the buffer window. Since `timedelta`
already exists for users who want a grace window, `False` is cleaned up to strictly mean
what it says: always skip the past cron point.
**Why the default changed from `False` to `True`**
Because of the bug, `run_immediately=False` was silently ignored whenever `start_date` was
set. Since almost every production DAG sets `start_date`, the default `False` always behaved
like `True` for the vast majority of users in practice. Changing the default to `True` preserves
what users have always observed. If we kept the default as `False` and only fixed the bug,
every existing DAG without an explicit `run_immediately` would suddenly stop running the past
cron point on first enable — a much larger breaking change.
> I think this PR is mixing a legitimate bug fix i.e. ensuring that setting the start_date does not subvert run_immediately in the timetable logic, with a behavioral change that feels a bit opinionated. Always routing the first run through _calc_first_run() makes sense, since previously run_immediately could effectively be ignored when start_date was set which is clearly inconsistent.
This is my actual pain point. I have updated the description as stated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-4119703840
| `start_date` | `run_immediately` | Before this PR | After this PR |
|---|---|---|---|
| set | default (`True`) | Old default was `False`; Ran past boundary immediately | Runs past boundary immediately |
| set | explicit `True` | Ran past boundary immediately | Runs past boundary immediately |
| set | explicit `False` | Ran past boundary immediately| Waits for next future boundary |
| `None` | default (`True`) | Old default was `False`; waited for next future boundary | Runs past boundary immediately |
| `None` | explicit `True` | Ran past boundary immediately | Runs past boundary immediately |
| `None` | explicit `False` | Waited for next future boundary | Waits for next future boundary |
cc: @collinmcnulty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3578845533
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -186,7 +187,7 @@ def __init__(
*,
timezone: str | Timezone | FixedTimezone,
interval: datetime.timedelta | relativedelta = datetime.timedelta(),
- run_immediately: bool | datetime.timedelta = False,
+ run_immediately: bool | datetime.timedelta = True,
Review Comment:
I have modified the default value so it can keep up with the existing behaviour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "manipatnam (via GitHub)" <gi...@apache.org>.
manipatnam commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3578421162
##########
airflow-core/docs/authoring-and-scheduling/timetable.rst:
##########
@@ -151,6 +151,63 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
def example_dag():
pass
+.. versionadded:: 3.0.0
+ The ``run_immediately`` argument was introduced in Airflow 3.
+
+The optional ``run_immediately`` argument controls which cron point is scheduled when a Dag is first
+enabled or re-enabled after a pause. It has no effect when ``catchup=True`` (in that case the
+scheduler always continues from where it left off).
+
+* ``run_immediately=True`` *(default)* — schedule the **most recent past** cron point immediately.
+* ``run_immediately=False`` — skip the past cron point and wait for the **next future** cron point.
+* ``run_immediately=timedelta(...)`` — schedule the most recent past cron point only if it fired
+ within the given window; otherwise wait for the **next future** cron point.
+
+.. code-block:: python
+
+ from datetime import datetime, timedelta
+
+ from airflow.timetables.trigger import CronTriggerTimetable
+
+
+ @dag(
+ # Runs every 10 minutes.
+ # run_immediately=False: always skip the most recent past slot and wait
+ # for the next 10-minute boundary.
+ schedule=CronTriggerTimetable(
+ "*/10 * * * *",
+ timezone="UTC",
+ run_immediately=False,
+ ),
+ start_date=datetime(2024, 1, 1),
+ catchup=False,
+ ...,
+ )
+ def example_dag():
+ pass
+
+
+ @dag(
+ # Runs hourly.
+ # run_immediately=timedelta(minutes=10): run the most recent past slot
+ # only if it fired within the last 10 minutes; otherwise wait for next.
+ schedule=CronTriggerTimetable(
+ "0 * * * *",
+ timezone="UTC",
+ run_immediately=timedelta(minutes=10),
+ ),
+ start_date=datetime(2024, 1, 1),
+ catchup=False,
+ ...,
+ )
+ def example_dag_with_buffer():
+ pass
+
+.. note::
+
+ ``run_immediately`` is a parameter of ``CronTriggerTimetable``, **not** of the ``DAG``
+ constructor. Passing it directly to ``DAG(run_immediately=...)`` has no effect.
Review Comment:
Thanks for highlighting that the earlier wording was wrong. I have corrected it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-4439081438
LGTM! I still prefer more pair of eyes before merging this. @uranusjr can you review this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-5102717954
I see few unresolved conversation and few conflicts might need work. Moving to 3.3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #64162:
URL: https://github.com/apache/airflow/pull/64162#issuecomment-4947260120
@manipatnam I've removed the `ready for maintainer review` label because the next step here is yours: unresolved review thread(s) from `@uranusjr` awaiting your reply. It'll go back into the maintainer queue automatically once that's done — no need to re-add the label by hand. Next move: address the unresolved threads and reply to `@uranusjr`. There is no rush.
---
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. If you have questions, an Apache Airflow maintainer — a real person — is happy to help. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhanced run_immediately functionality [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #64162:
URL: https://github.com/apache/airflow/pull/64162#discussion_r3457504772
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -87,12 +87,15 @@ def next_dagrun_info(
else:
next_start_time = self._align_to_next(restriction.earliest)
else:
- start_time_candidates = [self._align_to_prev(coerce_datetime(utcnow()))]
if last_automated_data_interval is not None:
- start_time_candidates.append(self._get_next(last_automated_data_interval.end))
- elif restriction.earliest is None:
- # Run immediately has no effect if there is restriction on earliest
- start_time_candidates.append(self._calc_first_run())
+ # _calc_first_run respects run_immediately to decide between the
+ # most recent past cron point and the next future one.
Review Comment:
What is this comment trying to explain? `run_immediately` is not mentioned anywhere around this line. Why does it need to be mentioned here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org