You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "seanghaeli (via GitHub)" <gi...@apache.org> on 2026/06/23 20:17:20 UTC
[PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
seanghaeli opened a new pull request, #68917:
URL: https://github.com/apache/airflow/pull/68917
## Why
This re-introduces the **`VariableInterval` resolution** portion of #66608, which was fully reverted in #68909. That revert removed several unrelated concerns (callback context delivery, an execution-API token-scope change flagged as a security regression, UI work). This PR brings back only the self-contained, security-neutral piece: safely resolving a `VariableInterval`-backed deadline at DagRun creation.
A `DeadlineAlert` configured with a `VariableInterval` is resolved when the scheduler creates a DagRun, inside `DAG._process_dagrun_deadline_alerts` (which runs under the `prohibit_commit` guard). Two problems are fixed:
1. **Resolution goes through the full secrets chain** (env vars, configured secrets backends, then the metadata DB) via a dedicated `_resolve_variable_interval` helper, rather than `Variable.get` / `begin_nested`. `Variable.get` and a SAVEPOINT release both commit on the scheduler's session, tripping `prohibit_commit` (`UNEXPECTED COMMIT`) and silently dropping deadlines for every scheduled DagRun. The helper passes the scheduler session through to the metastore backend so the DB read does not commit, and reading via the secrets chain (not the `variable` table directly) means `AIRFLOW_VAR_*` env vars and secrets-backend-backed Variables resolve too.
2. **Each deadline alert is isolated** with a plain `try`/`except` (deliberately **not** `begin_nested`, which would commit a SAVEPOINT and trip the same guard). Creating a deadline is auxiliary to creating the DagRun; a single bad alert — a missing/invalid backing Variable, or an undecodable serialized blob — must never abort the DagRun and stop the DAG from scheduling.
`VariableInterval.resolve` is split into `resolve` + `coerce_to_timedelta` so the scheduler-side reader reuses the exact same validation (including the `OverflowError` -> `ValueError` translation) without going through `Variable.get`.
## Tests
- `airflow-core/tests/unit/models/test_dagrun.py`: VariableInterval resolves from a real Variable row and from an `AIRFLOW_VAR_*` env var; a missing Variable and an undecodable alert are isolated (DagRun still created, no Deadline row, error logged).
- `task-sdk/tests/task_sdk/definitions/test_deadline.py`: `coerce_to_timedelta` validation (non-integer, `<= 0`, overflow).
Verified locally in Breeze: 8 passed (dagrun deadline/variable) + 14 passed (SDK `TestVariableInterval`).
Generated-by: Claude Code (Opus via Claude Code) on behalf of Sean Ghaeli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r4021331168
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1610,30 +1610,136 @@ def test_dagrun_deadline_does_not_warn_for_average_runtime_without_history(
"log_level": "warning",
} not in caplog
+ @pytest.mark.parametrize(
+ ("interval", "failure"),
+ [
+ pytest.param(
+ VariableInterval("missing_key"),
+ mock.patch.object(Variable, "get", side_effect=KeyError),
+ id="unresolvable_interval",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch(
+ "airflow.serialization.definitions.dag.decode_deadline_alert",
+ autospec=True,
+ side_effect=ValueError("corrupt deadline alert blob"),
+ ),
+ id="undecodable_alert",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch.object(
+ SerializedReferenceModels.FixedDatetimeDeadline,
+ "evaluate_with",
+ autospec=True,
+ side_effect=RuntimeError("evaluate_with failed"),
+ ),
+ id="unevaluable_reference",
+ ),
+ ],
+ )
+ @mock.patch("airflow._shared.observability.metrics.stats.incr")
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_missing_variable_fails(self, _, session, deadline_test_dag):
+ def test_dagrun_deadline_failure_does_not_abort_dagrun(
+ self, _, mock_stats_incr, interval, failure, session, deadline_test_dag
+ ):
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
- with mock.patch.object(
- Variable,
- "get",
- side_effect=KeyError,
+ scheduler_dag = deadline_test_dag(
+ deadline=DeadlineAlert(
+ reference=DeadlineReference.FIXED_DATETIME(future_date),
+ interval=interval,
+ callback=AsyncCallback(empty_callback_for_deadline),
+ ),
+ )
+
+ with (
+ conf_vars({("core", "multi_team"): "true"}),
+ mock.patch("airflow.models.dag.DagModel.get_team_name", return_value="team_alpha"),
+ failure,
):
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
+ dag_run = self.create_dag_run(
+ dag=scheduler_dag,
+ task_states={"task_1": TaskInstanceState.SUCCESS},
+ session=session,
+ )
+
+ assert dag_run is not None
+ assert session.execute(select(Deadline)).scalars().one_or_none() is None
+ mock_stats_incr.assert_any_call(
+ "deadline_alerts.deadline_creation_failed",
+ tags={"dag_id": scheduler_dag.dag_id, "team_name": "team_alpha"},
+ )
- scheduler_dag = deadline_test_dag(
- deadline=DeadlineAlert(
- reference=DeadlineReference.FIXED_DATETIME(future_date),
- interval=VariableInterval("missing_key"),
- callback=AsyncCallback(empty_callback_for_deadline),
+ @mock.patch("airflow._shared.observability.metrics.stats.incr")
+ @mock.patch.object(Deadline, "prune_deadlines")
+ @mock.patch.object(Variable, "get")
+ def test_dagrun_deadline_failure_that_detaches_orm_objects_still_skips(
+ self, mock_variable_get, _, mock_stats_incr, session, deadline_test_dag
+ ):
+ """A failing interval resolution can roll back and close the caller's session underneath
+ the handler (``create_session`` reusing the scoped session does exactly that), detaching
+ every ORM instance. The skip path must still log and count without touching ORM state."""
+ scheduler_dag = deadline_test_dag(
+ deadline=DeadlineAlert(
+ reference=DeadlineReference.FIXED_DATETIME(
+ datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
),
- )
+ interval=VariableInterval("missing_key"),
+ callback=AsyncCallback(empty_callback_for_deadline),
+ ),
+ )
+ # The alert rows must be persistent from an earlier transaction (as in the live
+ # scheduler, where the dag processor committed them) so the rollback below expires
+ # them and the close detaches them instead of returning them to transient.
+ session.commit()
- with pytest.raises(ValueError, match="not found"):
- self.create_dag_run(
- dag=scheduler_dag,
- task_states={"task_1": TaskInstanceState.SUCCESS},
- session=session,
- )
+ def teardown_session_and_raise(*args, **kwargs):
+ session.rollback()
+ session.close()
+ raise KeyError("missing_key")
+
+ mock_variable_get.side_effect = teardown_session_and_raise
+
+ dag_run = self.create_dag_run(dag=scheduler_dag, session=session)
+
+ assert dag_run is not None
+ mock_stats_incr.assert_any_call(
+ "deadline_alerts.deadline_creation_failed",
+ tags={"dag_id": scheduler_dag.dag_id},
+ )
+
+ @pytest.mark.parametrize(
+ ("interval", "expect_deadline"),
+ [
+ pytest.param(datetime.timedelta(seconds=5), True, id="deadline_created"),
+ pytest.param(VariableInterval("missing_key"), False, id="deadline_skipped"),
+ ],
+ )
+ @mock.patch.object(Deadline, "prune_deadlines")
+ def test_dagrun_deadline_handling_does_not_commit(
Review Comment:
Added this + test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3787932135
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,121 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
+ @pytest.mark.parametrize(
+ ("interval", "failure"),
+ [
+ pytest.param(VariableInterval("missing_key"), nullcontext(), id="missing_variable"),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch(
+ "airflow.serialization.definitions.dag.decode_deadline_alert",
+ autospec=True,
+ side_effect=ValueError("corrupt deadline alert blob"),
+ ),
+ id="decode_failure",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch.object(
+ SerializedReferenceModels.FixedDatetimeDeadline,
+ "evaluate_with",
+ autospec=True,
+ side_effect=RuntimeError("evaluate_with failed"),
+ ),
+ id="evaluate_with_failure",
+ ),
+ ],
+ )
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
+ def test_dagrun_deadline_failure_is_isolated(self, _, interval, failure, session, deadline_test_dag):
+ """A failure while creating any single deadline must not abort DagRun creation."""
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
+
+ scheduler_dag = deadline_test_dag(
+ deadline=DeadlineAlert(
+ reference=DeadlineReference.FIXED_DATETIME(future_date),
+ interval=interval,
+ callback=AsyncCallback(empty_callback_for_deadline),
+ ),
+ )
+
+ with failure:
+ dag_run = self.create_dag_run(
+ dag=scheduler_dag,
+ task_states={"task_1": TaskInstanceState.SUCCESS},
+ session=session,
+ )
- # First value used during resolution.
- mock_get.return_value = "60"
+ assert dag_run is not None
+ assert session.execute(select(Deadline)).scalars().one_or_none() is None
+
+ @mock.patch.object(Deadline, "prune_deadlines")
+ def test_dagrun_deadline_variable_interval_resolves_from_env_var(
+ self, _, session, deadline_test_dag, monkeypatch
+ ):
+ """A VariableInterval backed by an ``AIRFLOW_VAR_*`` env var (no DB row) must resolve."""
+ monkeypatch.setenv("AIRFLOW_VAR_ENV_INTERVAL_KEY", "7")
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
scheduler_dag = deadline_test_dag(
deadline=DeadlineAlert(
reference=DeadlineReference.FIXED_DATETIME(future_date),
- interval=VariableInterval("my_key"),
+ interval=VariableInterval("env_interval_key"),
callback=AsyncCallback(empty_callback_for_deadline),
),
)
dag_run = self.create_dag_run(
dag=scheduler_dag,
- task_states={"task_1": TaskInstanceState.SUCCESS, "task_2": TaskInstanceState.SUCCESS},
+ task_states={"task_1": TaskInstanceState.SUCCESS},
session=session,
)
- dag_run.dag = scheduler_dag
-
- # First update resolve interval to "5".
- dag_run.update_state(session=session)
-
- deadline = session.execute(select(Deadline)).scalars().one_or_none()
- first_deadline_time = deadline.deadline_time
-
- # Change Variable value after resolution.
- mock_get.return_value = "120"
-
- # Run again (This should not change existing deadline).
- dag_run.update_state(session=session)
+ assert dag_run is not None
deadline = session.execute(select(Deadline)).scalars().one_or_none()
- assert deadline.deadline_time == first_deadline_time
+ assert deadline is not None
+ assert deadline.deadline_time == future_date + datetime.timedelta(seconds=7)
+ @pytest.mark.parametrize(
+ ("multi_team", "team_name"),
+ [
+ pytest.param("true", "team_alpha", id="team_scoped"),
+ pytest.param("false", None, id="global"),
+ ],
+ )
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_missing_variable_fails(self, _, session, deadline_test_dag):
- mock_err = mock.Mock()
- mock_err.error.value = "MISSING_DEADLINE"
- mock_err.detail = "missing deadline"
+ def test_dagrun_deadline_variable_interval_scoped_to_team(
+ self, _, multi_team, team_name, session, deadline_test_dag
+ ):
+ """A VariableInterval must resolve against the DagRun's team, not the global scope."""
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
- with mock.patch.object(
- Variable,
- "get",
- side_effect=AirflowRuntimeError(mock_err),
- ):
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
+ scheduler_dag = deadline_test_dag(
+ deadline=DeadlineAlert(
+ reference=DeadlineReference.FIXED_DATETIME(future_date),
+ interval=VariableInterval("team_interval_key"),
+ callback=AsyncCallback(empty_callback_for_deadline),
+ ),
+ )
- scheduler_dag = deadline_test_dag(
- deadline=DeadlineAlert(
- reference=DeadlineReference.FIXED_DATETIME(future_date),
- interval=VariableInterval("missing_key"),
- callback=AsyncCallback(empty_callback_for_deadline),
- ),
+ with (
+ conf_vars({("core", "multi_team"): multi_team}),
+ mock.patch("airflow.models.dag.DagModel.get_team_name", return_value=team_name),
+ mock.patch(
+ "airflow.serialization.definitions.dag.call_secrets_backend_method",
+ return_value="5",
+ ) as mock_call,
+ ):
+ dag_run = self.create_dag_run(
+ dag=scheduler_dag,
+ task_states={"task_1": TaskInstanceState.SUCCESS},
+ session=session,
)
- with pytest.raises(ValueError, match="not found"):
- self.create_dag_run(
- dag=scheduler_dag,
- task_states={"task_1": TaskInstanceState.SUCCESS},
- session=session,
- )
+ assert dag_run is not None
+ # The team the DagRun belongs to must be forwarded to the backend lookup.
+ mock_call.assert_called_once()
+ assert mock_call.call_args.kwargs["team_name"] == team_name
Review Comment:
Good catch, definitely an oversight by me. Addressed in the latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68917:
URL: https://github.com/apache/airflow/pull/68917#issuecomment-6247959455
### Backport successfully created: v3-4-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-4-test</td>
<td><a href="https://github.com/apache/airflow/pull/74975"><img src="https://img.shields.io/badge/PR-74975-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3807992172
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,52 +743,100 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
-
- interval = deserialized_deadline_alert.interval
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ interval = deserialized_deadline_alert.interval
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
+ # Resolve the DagRun's team once, so a team-scoped VariableInterval is looked up
+ # against the right team (not the global scope) and the stats tag is consistent.
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
)
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
- )
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, team_name=team_name, session=session)
+
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
)
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(
+ interval: VariableInterval, *, team_name: str | None, session: Session
+ ) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
+
+ The Variable is resolved using the standard secrets lookup order. The scheduler
+ session is passed to the metastore backend to avoid creating a new session
+ during DagRun creation.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param team_name: Team owning the DagRun, forwarded to scope the Variable lookup.
+ :param session: Scheduler session used for metadata database lookups.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+ """
+ for backend in ensure_secrets_loaded():
Review Comment:
I agree. The only difference between core's `Variable.get_variable_from_secrets` and ours is the addition of:
`**({"session": session} if isinstance(backend, MetastoreBackend) else {})`
Which I've opted to avoid adding to core because it would be a much wider change if we altered the core implementation. So, I agree, I've made an issue to track this in a future PR here: https://github.com/apache/airflow/issues/71801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3817353729
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,121 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
+ @pytest.mark.parametrize(
+ ("interval", "failure"),
+ [
+ pytest.param(VariableInterval("missing_key"), nullcontext(), id="missing_variable"),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch(
+ "airflow.serialization.definitions.dag.decode_deadline_alert",
+ autospec=True,
+ side_effect=ValueError("corrupt deadline alert blob"),
+ ),
+ id="decode_failure",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch.object(
+ SerializedReferenceModels.FixedDatetimeDeadline,
+ "evaluate_with",
+ autospec=True,
+ side_effect=RuntimeError("evaluate_with failed"),
+ ),
+ id="evaluate_with_failure",
+ ),
+ ],
+ )
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
Review Comment:
Likely not mid-run. I think, the best I can come up with is if a run with a queued_at reference is cleared, but that is covered elsewhere. I think we're likely good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3508707473
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
Review Comment:
I think you need to adjust the AI tool you are using to avoid generating these massive comments and docstrings. I would suggest the below instead:
```
# Deadline creation is best-effort. A failure here must not prevent the DagRun
# itself from being created. Use a plain try/except rather than
# ``session.begin_nested()`` since ``create_dagrun`` runs under
# ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
```
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
Review Comment:
This except is too broad. If the only errors that you expect to encounter here are likely to do with `_resolve_variable_interval` or deserialization, why not limit it to them? Maybe something like this would be better:
`except (ValueError, DeserializationError):`
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(interval: VariableInterval, *, session: Session) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation time.
+
+ The backing Variable is read through the full secrets chain (env vars, configured
+ secrets backends, then the metadata DB) -- the same resolution order as
+ ``Variable.get`` -- rather than reading only the ``variable`` table. Reading the table
+ directly would bypass ``AIRFLOW_VAR_*`` env vars and secrets backends, so a Variable
+ that lives there would resolve to ``None`` and the deadline would be silently dropped.
+
+ We must NOT call ``Variable.get`` / ``get_variable_from_secrets`` here: the metastore
+ backend's ``get_variable`` is ``@provide_session`` and, given no session, opens the
+ thread-local scoped session (the SAME one the scheduler holds) whose context-manager
+ exit COMMITS -- tripping the ``prohibit_commit`` guard ``create_dagrun`` runs under.
+ Instead we iterate the backends ourselves and pass the scheduler's ``session`` through
+ to the metastore backend (env / custom backends ignore it), so the DB read happens on
+ our session with no commit.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param session: The scheduler session, passed through to the metastore backend.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be found in any backend, or its value
+ cannot be coerced to a positive ``timedelta``.
+ """
+ from airflow._shared.secrets_backend.base import call_secrets_backend_method
+ from airflow.configuration import ensure_secrets_loaded
+ from airflow.secrets.metastore import MetastoreBackend
Review Comment:
Do these imports need to be here to avoid circular imports or something like that? If not, I would move them to the top.
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(interval: VariableInterval, *, session: Session) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation time.
+
+ The backing Variable is read through the full secrets chain (env vars, configured
+ secrets backends, then the metadata DB) -- the same resolution order as
+ ``Variable.get`` -- rather than reading only the ``variable`` table. Reading the table
+ directly would bypass ``AIRFLOW_VAR_*`` env vars and secrets backends, so a Variable
+ that lives there would resolve to ``None`` and the deadline would be silently dropped.
+
+ We must NOT call ``Variable.get`` / ``get_variable_from_secrets`` here: the metastore
+ backend's ``get_variable`` is ``@provide_session`` and, given no session, opens the
+ thread-local scoped session (the SAME one the scheduler holds) whose context-manager
+ exit COMMITS -- tripping the ``prohibit_commit`` guard ``create_dagrun`` runs under.
+ Instead we iterate the backends ourselves and pass the scheduler's ``session`` through
+ to the metastore backend (env / custom backends ignore it), so the DB read happens on
+ our session with no commit.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param session: The scheduler session, passed through to the metastore backend.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be found in any backend, or its value
+ cannot be coerced to a positive ``timedelta``.
+ """
Review Comment:
Same issue as above. I would suggest the below:
"""
Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
The Variable is resolved using the standard secrets lookup order. The scheduler
session is passed to the metastore backend to avoid creating a new session
during DagRun creation.
:param interval: The ``VariableInterval`` to resolve.
:param session: Scheduler session used for metadata database lookups.
:return: The resolved ``timedelta``.
:raises ValueError: If the Variable cannot be resolved or converted to a valid
``timedelta``.
"""
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(interval: VariableInterval, *, session: Session) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation time.
+
+ The backing Variable is read through the full secrets chain (env vars, configured
+ secrets backends, then the metadata DB) -- the same resolution order as
+ ``Variable.get`` -- rather than reading only the ``variable`` table. Reading the table
+ directly would bypass ``AIRFLOW_VAR_*`` env vars and secrets backends, so a Variable
+ that lives there would resolve to ``None`` and the deadline would be silently dropped.
+
+ We must NOT call ``Variable.get`` / ``get_variable_from_secrets`` here: the metastore
+ backend's ``get_variable`` is ``@provide_session`` and, given no session, opens the
+ thread-local scoped session (the SAME one the scheduler holds) whose context-manager
+ exit COMMITS -- tripping the ``prohibit_commit`` guard ``create_dagrun`` runs under.
+ Instead we iterate the backends ourselves and pass the scheduler's ``session`` through
+ to the metastore backend (env / custom backends ignore it), so the DB read happens on
+ our session with no commit.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param session: The scheduler session, passed through to the metastore backend.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be found in any backend, or its value
+ cannot be coerced to a positive ``timedelta``.
+ """
+ from airflow._shared.secrets_backend.base import call_secrets_backend_method
+ from airflow.configuration import ensure_secrets_loaded
+ from airflow.secrets.metastore import MetastoreBackend
+
+ var_val = None
+ for secrets_backend in ensure_secrets_loaded():
+ kwargs = {"session": session} if isinstance(secrets_backend, MetastoreBackend) else {}
+ var_val = call_secrets_backend_method(
+ secrets_backend.get_variable, team_name=None, key=interval.key, **kwargs
+ )
+ if var_val is not None:
+ break
+
+ if var_val is None:
+ # Fail loudly (the per-alert ``except`` isolates it): the Variable does not exist in
+ # any backend, so we cannot resolve the interval. This is not a silent skip.
+ raise ValueError(
+ f"VariableInterval '{interval.key}' could not be resolved from any "
+ f"secrets backend, environment variable, or the metadata database"
+ )
+
+ return interval.coerce_to_timedelta(var_val)
Review Comment:
This could be much cleaner. Please see the below:
```
for backend in ensure_secrets_loaded():
value = call_secrets_backend_method(
backend.get_variable,
team_name=None,
key=interval.key,
**({"session": session} if isinstance(backend, MetastoreBackend) else {}),
)
if value is not None:
return interval.coerce_to_timedelta(value)
# Fail loudly (the per-alert ``except`` isolates it): the Variable does not exist in
# any backend, so we cannot resolve the interval. This is not a silent skip.
raise ValueError(
f"VariableInterval '{interval.key}' could not be resolved from any "
f"secrets backend, environment variable, or the metadata database"
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3788093638
##########
task-sdk/src/airflow/sdk/definitions/deadline.py:
##########
@@ -423,9 +423,19 @@ def resolve(self) -> timedelta:
value = Variable.get(self.key)
except AirflowRuntimeError as e:
raise ValueError(f"VariableInterval '{self.key}' not found") from e
+ return self.coerce_to_timedelta(value)
+ def coerce_to_timedelta(self, value: str | int | float | None) -> timedelta:
+ """
Review Comment:
Updated to follow your docstring suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3884902298
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -737,55 +740,118 @@ def _process_dagrun_deadline_alerts(
select(DeadlineAlertModel).where(DeadlineAlertModel.serialized_dag_id == serialized_dag_id)
).all()
+ if not deadline_alert_records:
+ return
+
+ # Resolve the DagRun's team once, so a team-scoped VariableInterval is looked up
+ # against the right team (not the global scope) and the stats tags are consistent.
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ metrics_tags = prune_dict({"dag_id": self.dag_id, "team_name": team_name})
+
for deadline_alert in deadline_alert_records:
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, team_name=team_name, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ stats.incr("deadline_alerts.deadline_created", tags=metrics_tags)
Review Comment:
You are going to end up having to rebase on top of a couple of other PRs that need to land before this one. I'm trying to make that future rebase a bit easier here.
#71767 has an elif branch right after the stats.incr you have on L800. If you can add that here, it should ease your merge pain later:
```python
stats.incr("deadline_alerts.deadline_created", tags=metrics_tags)
elif required_dagrun_column := {
SerializedReferenceModels.DagRunLogicalDateDeadline: "logical_date",
SerializedReferenceModels.DagRunQueuedAtDeadline: "queued_at",
}.get(type(deserialized_deadline_alert.reference)):
log.warning(
"skipping deadline alert because the deadline reference evaluated to None",
dag_id=self.dag_id,
run_id=orm_dagrun.run_id,
deadline_alert_id=deadline_alert.id,
reference_type=deserialized_deadline_alert.reference.reference_name,
required_dagrun_column=required_dagrun_column,
)
```
That PR also adds two tests in `airflow-core/tests/unit/models/test_dagrun.py` that you'll want to pull over which cover that branch. That can wait till the merge but may be easier to just paste it over now since you are copying this over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on PR #68917:
URL: https://github.com/apache/airflow/pull/68917#issuecomment-5689817942
@ferruzzi up to date with rebases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r4010294598
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
Review Comment:
I agree on principle but there's one hitch here. We need to re-raise the `RuntimeError` that a session collision can raise here. This is part of what I've been trying to untangle, but basically this code falls inside the scheduler's `prohibit_commit` guard which can raise `RuntimeError("UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!")` and we need that one specific message bubbling up or you'll be swallowing legitimate database safety issues.
It won't cost you the DagRun (which you rightly used to push back earlier) since the guard fires inside `session.commit()`, so by the time your `except` sees it, the scheduler has already rolled it back and closed the session. So in this case, the DagRun is already shot and we should capture why.
To be clear, this isn't something you could have necessarily seen since you forward session to both `resolve()` and `evaluate_with()`, but it becomes an issue once #70370 lands. I'm still (and I'm sorry for how long it's taking) just paving the way for easier rebases/merges later and you're next in the chain behind #70370.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r4010427616
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
Review Comment:
Sorry for the churn. I should have waited until i was done reviewing before I posted that. You already have a `prohibit_commit` test, so you likely already knew most of that. The request for reraising the RuntimeError stands, though.
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1610,30 +1610,136 @@ def test_dagrun_deadline_does_not_warn_for_average_runtime_without_history(
"log_level": "warning",
} not in caplog
+ @pytest.mark.parametrize(
+ ("interval", "failure"),
+ [
+ pytest.param(
+ VariableInterval("missing_key"),
+ mock.patch.object(Variable, "get", side_effect=KeyError),
+ id="unresolvable_interval",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch(
+ "airflow.serialization.definitions.dag.decode_deadline_alert",
+ autospec=True,
+ side_effect=ValueError("corrupt deadline alert blob"),
+ ),
+ id="undecodable_alert",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch.object(
+ SerializedReferenceModels.FixedDatetimeDeadline,
+ "evaluate_with",
+ autospec=True,
+ side_effect=RuntimeError("evaluate_with failed"),
+ ),
+ id="unevaluable_reference",
+ ),
+ ],
+ )
+ @mock.patch("airflow._shared.observability.metrics.stats.incr")
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_missing_variable_fails(self, _, session, deadline_test_dag):
+ def test_dagrun_deadline_failure_does_not_abort_dagrun(
+ self, _, mock_stats_incr, interval, failure, session, deadline_test_dag
+ ):
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
- with mock.patch.object(
- Variable,
- "get",
- side_effect=KeyError,
+ scheduler_dag = deadline_test_dag(
+ deadline=DeadlineAlert(
+ reference=DeadlineReference.FIXED_DATETIME(future_date),
+ interval=interval,
+ callback=AsyncCallback(empty_callback_for_deadline),
+ ),
+ )
+
+ with (
+ conf_vars({("core", "multi_team"): "true"}),
+ mock.patch("airflow.models.dag.DagModel.get_team_name", return_value="team_alpha"),
+ failure,
):
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
+ dag_run = self.create_dag_run(
+ dag=scheduler_dag,
+ task_states={"task_1": TaskInstanceState.SUCCESS},
+ session=session,
+ )
+
+ assert dag_run is not None
+ assert session.execute(select(Deadline)).scalars().one_or_none() is None
+ mock_stats_incr.assert_any_call(
+ "deadline_alerts.deadline_creation_failed",
+ tags={"dag_id": scheduler_dag.dag_id, "team_name": "team_alpha"},
+ )
- scheduler_dag = deadline_test_dag(
- deadline=DeadlineAlert(
- reference=DeadlineReference.FIXED_DATETIME(future_date),
- interval=VariableInterval("missing_key"),
- callback=AsyncCallback(empty_callback_for_deadline),
+ @mock.patch("airflow._shared.observability.metrics.stats.incr")
+ @mock.patch.object(Deadline, "prune_deadlines")
+ @mock.patch.object(Variable, "get")
+ def test_dagrun_deadline_failure_that_detaches_orm_objects_still_skips(
+ self, mock_variable_get, _, mock_stats_incr, session, deadline_test_dag
+ ):
+ """A failing interval resolution can roll back and close the caller's session underneath
+ the handler (``create_session`` reusing the scoped session does exactly that), detaching
+ every ORM instance. The skip path must still log and count without touching ORM state."""
+ scheduler_dag = deadline_test_dag(
+ deadline=DeadlineAlert(
+ reference=DeadlineReference.FIXED_DATETIME(
+ datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
),
- )
+ interval=VariableInterval("missing_key"),
+ callback=AsyncCallback(empty_callback_for_deadline),
+ ),
+ )
+ # The alert rows must be persistent from an earlier transaction (as in the live
+ # scheduler, where the dag processor committed them) so the rollback below expires
+ # them and the close detaches them instead of returning them to transient.
+ session.commit()
- with pytest.raises(ValueError, match="not found"):
- self.create_dag_run(
- dag=scheduler_dag,
- task_states={"task_1": TaskInstanceState.SUCCESS},
- session=session,
- )
+ def teardown_session_and_raise(*args, **kwargs):
+ session.rollback()
+ session.close()
+ raise KeyError("missing_key")
+
+ mock_variable_get.side_effect = teardown_session_and_raise
+
+ dag_run = self.create_dag_run(dag=scheduler_dag, session=session)
+
+ assert dag_run is not None
+ mock_stats_incr.assert_any_call(
+ "deadline_alerts.deadline_creation_failed",
+ tags={"dag_id": scheduler_dag.dag_id},
+ )
+
+ @pytest.mark.parametrize(
+ ("interval", "expect_deadline"),
+ [
+ pytest.param(datetime.timedelta(seconds=5), True, id="deadline_created"),
+ pytest.param(VariableInterval("missing_key"), False, id="deadline_skipped"),
+ ],
+ )
+ @mock.patch.object(Deadline, "prune_deadlines")
+ def test_dagrun_deadline_handling_does_not_commit(
Review Comment:
This test almost catches the problem in the other thread, and I think one more case would be great.
Both of these param cases avoid resolving a `VariableInterval` successfully under the guard. `deadline_created` uses a plain `timedelta` so `resolve()` is never called, and `deadline_skipped` has `Variable.get` raise a `KeyError` before anything can commit. Since the `mock.patch.object(Variable, "get", side_effect=KeyError)` on line 1736 applies to both, there is no path here where the `Variable` lookup actually succeeds inside `prohibit_commit`.
Can you add a third case along the lines of `pytest.param(VariableInterval("some_key"), True, id="variable_interval_resolved")` with `Variable.get` returning some stringified integer ("4200" or whatever) rather than raising? That should hit the session-collision section and would fail if anyone drops the `session` forwarding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3798272845
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,52 +744,112 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
-
- interval = deserialized_deadline_alert.interval
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ interval = deserialized_deadline_alert.interval
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
+ # Resolve the DagRun's team once, so a team-scoped VariableInterval is looked up
+ # against the right team (not the global scope) and the stats tag is consistent.
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
)
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
- )
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, team_name=team_name, session=session)
+
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
)
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
Review Comment:
Nit: You have another metric `deadline_creation_failed` down on L808 which isn't getting team_name. Maybe build a `metrics_tags = prune_dict({"dag_id": self.dag_id, "team_name": team_name}` up above and reuse it for both, or add `team_name` to the other one directly?
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1390,16 +1388,20 @@ def test_dag_run_dag_versions_with_null_created_dag_version(self, dag_maker, ses
VariableInterval("my_key"),
],
)
- @mock.patch.object(Variable, "get")
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_success_deadline(self, _, mock_get, interval, session, deadline_test_dag):
+ def test_dagrun_success_deadline(self, _, interval, session, deadline_test_dag):
def on_success_callable(context):
assert context["dag_run"].dag_id == "test_dag"
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
- # First value used during resolution
- mock_get.return_value = "5"
+ if isinstance(interval, VariableInterval):
+ # Seed via the metastore model, not the SDK Variable (whose set() routes through
+ # SUPERVISOR_COMMS), so the row lands in the variable table on this session.
+ from airflow.models.variable import Variable as VariableModel
Review Comment:
Looks like this (and other) local imports can be top-level??
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,52 +743,100 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
-
- interval = deserialized_deadline_alert.interval
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ interval = deserialized_deadline_alert.interval
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
+ # Resolve the DagRun's team once, so a team-scoped VariableInterval is looked up
+ # against the right team (not the global scope) and the stats tag is consistent.
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
)
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
- )
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, team_name=team_name, session=session)
+
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
)
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(
+ interval: VariableInterval, *, team_name: str | None, session: Session
+ ) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
+
+ The Variable is resolved using the standard secrets lookup order. The scheduler
+ session is passed to the metastore backend to avoid creating a new session
+ during DagRun creation.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param team_name: Team owning the DagRun, forwarded to scope the Variable lookup.
+ :param session: Scheduler session used for metadata database lookups.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+ """
+ for backend in ensure_secrets_loaded():
Review Comment:
Sounds good. At this point it looks like it may be smarter to plumb `session` into `Variable.get_variable_from_secrets` since we've basically duplicated it here, but maybe you can just open an Issue to clean that up instead of holding this PR up longer over that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3762053839
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,51 +743,93 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(interval: VariableInterval, *, session: Session) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
+
+ The Variable is resolved using the standard secrets lookup order. The scheduler
+ session is passed to the metastore backend to avoid creating a new session
+ during DagRun creation.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param session: Scheduler session used for metadata database lookups.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+ """
+ for backend in ensure_secrets_loaded():
+ value = call_secrets_backend_method(
+ backend.get_variable,
+ team_name=None,
Review Comment:
Good catch, now the `team_name` is computed when multi team is one.
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,51 +743,93 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(interval: VariableInterval, *, session: Session) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
+
+ The Variable is resolved using the standard secrets lookup order. The scheduler
+ session is passed to the metastore backend to avoid creating a new session
+ during DagRun creation.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param session: Scheduler session used for metadata database lookups.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+ """
+ for backend in ensure_secrets_loaded():
+ value = call_secrets_backend_method(
+ backend.get_variable,
+ team_name=None,
Review Comment:
Good catch, now the `team_name` is computed when multi team is on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on PR #68917:
URL: https://github.com/apache/airflow/pull/68917#issuecomment-5639190899
@ferruzzi everything is up to date now, rebased and updated based on the other PRs that have been merged. I've updated the PR title and description to reflect the new scope.
Also, there is no collision with https://github.com/apache/airflow/pull/71287, so no concern there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3716660303
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,51 +743,93 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
Review Comment:
Added a test for this, good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r4021322546
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
Review Comment:
Added to re-raise
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
Review Comment:
Added to re-raise
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
Review Comment:
Added the re-raise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3798214649
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,121 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
+ @pytest.mark.parametrize(
+ ("interval", "failure"),
+ [
+ pytest.param(VariableInterval("missing_key"), nullcontext(), id="missing_variable"),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch(
+ "airflow.serialization.definitions.dag.decode_deadline_alert",
+ autospec=True,
+ side_effect=ValueError("corrupt deadline alert blob"),
+ ),
+ id="decode_failure",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch.object(
+ SerializedReferenceModels.FixedDatetimeDeadline,
+ "evaluate_with",
+ autospec=True,
+ side_effect=RuntimeError("evaluate_with failed"),
+ ),
+ id="evaluate_with_failure",
+ ),
+ ],
+ )
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
Review Comment:
Ah, alright, I guess that makes sense, sorry. I saw the removed tests and didn't see anything that looked like a direct correlation in the new tests.
> "Perhaps this points to a gap in update_state, but that's something we can address in a separate issue."
Want to open an Issue so someone can look into that?
Feel free to resolve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on PR #68917:
URL: https://github.com/apache/airflow/pull/68917#issuecomment-6090750999
Going to merge 70370 as soon as the CI goes green on that one, which unblocks this one. It should need another rebase to pull those changes in, just make sure when you do that you keep any places where session is being passed around; those are specific to fixing the Variable.get bug and can't get lost. If we did this right, then dropping it now should raise alarms, but before it would have been lost silently, which is why I was being so picky about the merge order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on PR #68917:
URL: https://github.com/apache/airflow/pull/68917#issuecomment-5111487989
@SameerMesiah97 could I get your inputs on the updated version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3787973008
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,52 +743,100 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
-
- interval = deserialized_deadline_alert.interval
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ interval = deserialized_deadline_alert.interval
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
+ # Resolve the DagRun's team once, so a team-scoped VariableInterval is looked up
+ # against the right team (not the global scope) and the stats tag is consistent.
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
)
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
- )
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, team_name=team_name, session=session)
+
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
)
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(
+ interval: VariableInterval, *, team_name: str | None, session: Session
+ ) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
+
+ The Variable is resolved using the standard secrets lookup order. The scheduler
+ session is passed to the metastore backend to avoid creating a new session
+ during DagRun creation.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param team_name: Team owning the DagRun, forwarded to scope the Variable lookup.
+ :param session: Scheduler session used for metadata database lookups.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+ """
+ for backend in ensure_secrets_loaded():
+ value = call_secrets_backend_method(
+ backend.get_variable,
+ team_name=team_name,
+ key=interval.key,
+ **({"session": session} if isinstance(backend, MetastoreBackend) else {}),
+ )
+ if value is not None:
+ return interval.coerce_to_timedelta(value)
Review Comment:
Nothing fetched here would be masked, so I don't believe there's any need to consider that here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3788087912
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,121 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
+ @pytest.mark.parametrize(
+ ("interval", "failure"),
+ [
+ pytest.param(VariableInterval("missing_key"), nullcontext(), id="missing_variable"),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch(
+ "airflow.serialization.definitions.dag.decode_deadline_alert",
+ autospec=True,
+ side_effect=ValueError("corrupt deadline alert blob"),
+ ),
+ id="decode_failure",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch.object(
+ SerializedReferenceModels.FixedDatetimeDeadline,
+ "evaluate_with",
+ autospec=True,
+ side_effect=RuntimeError("evaluate_with failed"),
+ ),
+ id="evaluate_with_failure",
+ ),
+ ],
+ )
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
Review Comment:
This PR rewrites the path that's tested here so it forced a decision about what to do with the test: delete it or update it. Upon further inspecting the test, I think it never asserted anything useful about deadlines to begin with, so I decided to delete it:
when the test does `dag_run.update_state(session)` it doesn't actually update the deadline. Therefore, the test can never possibly fail to begin with.
https://github.com/apache/airflow/blob/c74cfcbae17aa4aa89e66afca8e92035f957aba5/airflow-core/src/airflow/models/dagrun.py#L1371-L1381
In the source code for `update_state` the only logic for deadlines is pruning on success. Perhaps this points to a gap in `update_state`, but that's something we can address in a separate issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3807833711
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,121 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
+ @pytest.mark.parametrize(
+ ("interval", "failure"),
+ [
+ pytest.param(VariableInterval("missing_key"), nullcontext(), id="missing_variable"),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch(
+ "airflow.serialization.definitions.dag.decode_deadline_alert",
+ autospec=True,
+ side_effect=ValueError("corrupt deadline alert blob"),
+ ),
+ id="decode_failure",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch.object(
+ SerializedReferenceModels.FixedDatetimeDeadline,
+ "evaluate_with",
+ autospec=True,
+ side_effect=RuntimeError("evaluate_with failed"),
+ ),
+ id="evaluate_with_failure",
+ ),
+ ],
+ )
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
Review Comment:
It may not be a real gap, there's no real reason why a deadline would need to re-resolve mid-run. So, my instinct is to leave it out for the present day.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on PR #68917:
URL: https://github.com/apache/airflow/pull/68917#issuecomment-6153723080
@ferruzzi CI green now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3531936264
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
Review Comment:
Anything escaping would stop the DagRun being created, so I'd rather except broadly. There could be other failure modes so what makes sense to me is to except broadly and log the traceback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3532140680
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -715,51 +715,125 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Isolate each deadline alert: creating a deadline is auxiliary to creating the
+ # DagRun itself, and must never prevent the DagRun from being created. A single bad
+ # alert -- e.g. a ``VariableInterval`` whose backing Variable is missing / non-integer
+ # / <= 0 (``coerce_to_timedelta`` raises ``ValueError``), or a reference whose
+ # ``evaluate_with`` fails -- would otherwise propagate out of ``create_dagrun`` and
+ # abort the whole run, silently stopping the DAG from scheduling.
+ #
+ # Isolation is done with a plain ``try``/``except`` and MUST NOT use
+ # ``session.begin_nested()``: ``create_dagrun`` runs on the scheduler session inside
+ # the ``prohibit_commit`` guard, and releasing a SAVEPOINT issues a commit, which trips
+ # that guard (``RuntimeError("UNEXPECTED COMMIT ...")``) and -- because this very block
+ # swallows it -- silently skips deadline creation for *every* scheduled DagRun. The
+ # try/except alone is sufficient: the only DB mutation in the loop body is the final
+ # ``session.add`` (everything before it is a decode, an in-memory resolution, or a
+ # read-only ``evaluate_with`` query), so an exception leaves no partial state to undo,
+ # and the pending ``Deadline`` is persisted by the caller's outer transaction.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
Review Comment:
Okay. That makes perfects sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3695583135
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,51 +743,93 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
Review Comment:
Have you added a test for a scenario where `evaluate_with` fails or otherwise raises an error?
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,108 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
-
- # First value used during resolution.
- mock_get.return_value = "60"
+ def test_dagrun_deadline_variable_interval_missing_variable_is_isolated(
+ self, _, session, deadline_test_dag
+ ):
+ """A VariableInterval whose backing Variable is missing must NOT abort DagRun creation.
+
+ ``VariableInterval.resolve()`` raises ``ValueError`` for a missing/invalid Variable.
+ That resolution happens inside ``_process_dagrun_deadline_alerts`` during
+ ``create_dagrun``; previously the error propagated out and aborted the whole run,
+ silently stopping the DAG from scheduling. The per-alert ``try``/``except`` now isolates
+ the failure: the DagRun is created, the bad deadline is skipped (logged), and no Deadline
+ row is written. (Isolation must NOT use ``begin_nested`` here -- ``create_dagrun`` runs
+ under the scheduler ``prohibit_commit`` guard, where a SAVEPOINT release would raise
+ ``UNEXPECTED COMMIT`` and skip every scheduled DagRun's deadlines.)
+ """
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
scheduler_dag = deadline_test_dag(
deadline=DeadlineAlert(
reference=DeadlineReference.FIXED_DATETIME(future_date),
- interval=VariableInterval("my_key"),
+ interval=VariableInterval("missing_key"),
callback=AsyncCallback(empty_callback_for_deadline),
),
)
dag_run = self.create_dag_run(
dag=scheduler_dag,
- task_states={"task_1": TaskInstanceState.SUCCESS, "task_2": TaskInstanceState.SUCCESS},
+ task_states={"task_1": TaskInstanceState.SUCCESS},
session=session,
)
- dag_run.dag = scheduler_dag
-
- # First update resolve interval to "5".
- dag_run.update_state(session=session)
+ assert dag_run is not None
deadline = session.execute(select(Deadline)).scalars().one_or_none()
- first_deadline_time = deadline.deadline_time
+ assert deadline is None
- # Change Variable value after resolution.
- mock_get.return_value = "120"
+ @mock.patch.object(Deadline, "prune_deadlines")
+ def test_dagrun_deadline_variable_interval_resolves_from_env_var(
+ self, _, session, deadline_test_dag, monkeypatch
+ ):
+ """A VariableInterval backed by an ``AIRFLOW_VAR_*`` env var (no DB row) must resolve.
Review Comment:
The above feedback applies here too.
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,108 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
-
- # First value used during resolution.
- mock_get.return_value = "60"
+ def test_dagrun_deadline_variable_interval_missing_variable_is_isolated(
+ self, _, session, deadline_test_dag
+ ):
+ """A VariableInterval whose backing Variable is missing must NOT abort DagRun creation.
Review Comment:
Only keep the first line. Like I mentioned previously, we do not need long docstrings. Additional context should be covered in comments.
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,108 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
-
- # First value used during resolution.
- mock_get.return_value = "60"
+ def test_dagrun_deadline_variable_interval_missing_variable_is_isolated(
+ self, _, session, deadline_test_dag
+ ):
+ """A VariableInterval whose backing Variable is missing must NOT abort DagRun creation.
+
+ ``VariableInterval.resolve()`` raises ``ValueError`` for a missing/invalid Variable.
+ That resolution happens inside ``_process_dagrun_deadline_alerts`` during
+ ``create_dagrun``; previously the error propagated out and aborted the whole run,
+ silently stopping the DAG from scheduling. The per-alert ``try``/``except`` now isolates
+ the failure: the DagRun is created, the bad deadline is skipped (logged), and no Deadline
+ row is written. (Isolation must NOT use ``begin_nested`` here -- ``create_dagrun`` runs
+ under the scheduler ``prohibit_commit`` guard, where a SAVEPOINT release would raise
+ ``UNEXPECTED COMMIT`` and skip every scheduled DagRun's deadlines.)
+ """
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
scheduler_dag = deadline_test_dag(
deadline=DeadlineAlert(
reference=DeadlineReference.FIXED_DATETIME(future_date),
- interval=VariableInterval("my_key"),
+ interval=VariableInterval("missing_key"),
callback=AsyncCallback(empty_callback_for_deadline),
),
)
dag_run = self.create_dag_run(
dag=scheduler_dag,
- task_states={"task_1": TaskInstanceState.SUCCESS, "task_2": TaskInstanceState.SUCCESS},
+ task_states={"task_1": TaskInstanceState.SUCCESS},
session=session,
)
- dag_run.dag = scheduler_dag
-
- # First update resolve interval to "5".
- dag_run.update_state(session=session)
+ assert dag_run is not None
deadline = session.execute(select(Deadline)).scalars().one_or_none()
- first_deadline_time = deadline.deadline_time
+ assert deadline is None
- # Change Variable value after resolution.
- mock_get.return_value = "120"
+ @mock.patch.object(Deadline, "prune_deadlines")
+ def test_dagrun_deadline_variable_interval_resolves_from_env_var(
+ self, _, session, deadline_test_dag, monkeypatch
+ ):
+ """A VariableInterval backed by an ``AIRFLOW_VAR_*`` env var (no DB row) must resolve.
- # Run again (This should not change existing deadline).
- dag_run.update_state(session=session)
+ Regression guard: the scheduler-side resolver must go through the full secrets chain
+ (env vars + secrets backends + metadata DB), not read only the ``variable`` table. A
+ table-only read returns None for an env/secrets-backed Variable, and the per-alert
+ ``except`` then silently drops the deadline. Here the Variable lives ONLY in the
+ environment, so a correct resolver creates the Deadline and a regressed one drops it.
+ """
+ # Variable lives only as an env var, never in the variable table. Values are seconds.
+ monkeypatch.setenv("AIRFLOW_VAR_ENV_INTERVAL_KEY", "7")
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
+
+ scheduler_dag = deadline_test_dag(
+ deadline=DeadlineAlert(
+ reference=DeadlineReference.FIXED_DATETIME(future_date),
+ interval=VariableInterval("env_interval_key"),
+ callback=AsyncCallback(empty_callback_for_deadline),
+ ),
+ )
+
+ dag_run = self.create_dag_run(
+ dag=scheduler_dag,
+ task_states={"task_1": TaskInstanceState.SUCCESS},
+ session=session,
+ )
+ assert dag_run is not None
deadline = session.execute(select(Deadline)).scalars().one_or_none()
- assert deadline.deadline_time == first_deadline_time
+ assert deadline is not None
+ assert deadline.deadline_time == future_date + datetime.timedelta(seconds=7)
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_missing_variable_fails(self, _, session, deadline_test_dag):
- mock_err = mock.Mock()
- mock_err.error.value = "MISSING_DEADLINE"
- mock_err.detail = "missing deadline"
+ def test_dagrun_deadline_decode_failure_is_isolated(self, _, session, deadline_test_dag):
+ """A deadline alert that fails to *decode* must NOT abort DagRun creation either.
Review Comment:
Here too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3788135170
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,52 +743,100 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
-
- interval = deserialized_deadline_alert.interval
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ interval = deserialized_deadline_alert.interval
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
+ # Resolve the DagRun's team once, so a team-scoped VariableInterval is looked up
+ # against the right team (not the global scope) and the stats tag is consistent.
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
)
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
- )
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, team_name=team_name, session=session)
+
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
)
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(
+ interval: VariableInterval, *, team_name: str | None, session: Session
+ ) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
+
+ The Variable is resolved using the standard secrets lookup order. The scheduler
+ session is passed to the metastore backend to avoid creating a new session
+ during DagRun creation.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param team_name: Team owning the DagRun, forwarded to scope the Variable lookup.
+ :param session: Scheduler session used for metadata database lookups.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+ """
+ for backend in ensure_secrets_loaded():
Review Comment:
I believe there's no notion of caching in the scheduler. The analogous try/except blocks, I agree, we can add that. Added in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3777965163
##########
task-sdk/src/airflow/sdk/definitions/deadline.py:
##########
@@ -423,9 +423,19 @@ def resolve(self) -> timedelta:
value = Variable.get(self.key)
except AirflowRuntimeError as e:
raise ValueError(f"VariableInterval '{self.key}' not found") from e
+ return self.coerce_to_timedelta(value)
+ def coerce_to_timedelta(self, value: str | int | float | None) -> timedelta:
+ """
Review Comment:
Nit: Your docstring in SDK is explicitly referring to core internals, I don't think we're meant to be doing that. I'd just leave it as something like "Split out so a caller that already holds the value can reuse the validation" since that's all the SDK side knows.
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,121 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
+ @pytest.mark.parametrize(
+ ("interval", "failure"),
+ [
+ pytest.param(VariableInterval("missing_key"), nullcontext(), id="missing_variable"),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch(
+ "airflow.serialization.definitions.dag.decode_deadline_alert",
+ autospec=True,
+ side_effect=ValueError("corrupt deadline alert blob"),
+ ),
+ id="decode_failure",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch.object(
+ SerializedReferenceModels.FixedDatetimeDeadline,
+ "evaluate_with",
+ autospec=True,
+ side_effect=RuntimeError("evaluate_with failed"),
+ ),
+ id="evaluate_with_failure",
+ ),
+ ],
+ )
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
Review Comment:
Looks like a dropped test, is this covered by another or am I just misreading the diff and it's just a rename?
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,52 +743,100 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
-
- interval = deserialized_deadline_alert.interval
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ interval = deserialized_deadline_alert.interval
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
+ # Resolve the DagRun's team once, so a team-scoped VariableInterval is looked up
+ # against the right team (not the global scope) and the stats tag is consistent.
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
)
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
- )
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, team_name=team_name, session=session)
+
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
)
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(
+ interval: VariableInterval, *, team_name: str | None, session: Session
+ ) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
+
+ The Variable is resolved using the standard secrets lookup order. The scheduler
+ session is passed to the metastore backend to avoid creating a new session
+ during DagRun creation.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param team_name: Team owning the DagRun, forwarded to scope the Variable lookup.
+ :param session: Scheduler session used for metadata database lookups.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+ """
+ for backend in ensure_secrets_loaded():
Review Comment:
The core version of this (Variable.get_variable_from_secrets) has more logic like a tyr/except block and caching the returned value. Is there a reason we don't need any of that in here?
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,52 +743,100 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
-
- interval = deserialized_deadline_alert.interval
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ interval = deserialized_deadline_alert.interval
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
+ # Resolve the DagRun's team once, so a team-scoped VariableInterval is looked up
+ # against the right team (not the global scope) and the stats tag is consistent.
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
)
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
- )
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, team_name=team_name, session=session)
+
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
)
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(
+ interval: VariableInterval, *, team_name: str | None, session: Session
+ ) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
+
+ The Variable is resolved using the standard secrets lookup order. The scheduler
+ session is passed to the metastore backend to avoid creating a new session
+ during DagRun creation.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param team_name: Team owning the DagRun, forwarded to scope the Variable lookup.
+ :param session: Scheduler session used for metadata database lookups.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+ """
+ for backend in ensure_secrets_loaded():
+ value = call_secrets_backend_method(
+ backend.get_variable,
+ team_name=team_name,
+ key=interval.key,
+ **({"session": session} if isinstance(backend, MetastoreBackend) else {}),
+ )
+ if value is not None:
+ return interval.coerce_to_timedelta(value)
Review Comment:
We're using the secrets backend, do we need to mask or unmask here? In theory it's retrieving a number and that number itself may not be a secret, but I'm not positive if everything in there gets masked by default, etc.
##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -1509,71 +1511,121 @@ def test_dagrun_success_handles_empty_deadline_list(self, mock_prune, dag_maker,
mock_prune.assert_not_called()
assert dag_run.state == DagRunState.SUCCESS
- @mock.patch.object(Variable, "get")
+ @pytest.mark.parametrize(
+ ("interval", "failure"),
+ [
+ pytest.param(VariableInterval("missing_key"), nullcontext(), id="missing_variable"),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch(
+ "airflow.serialization.definitions.dag.decode_deadline_alert",
+ autospec=True,
+ side_effect=ValueError("corrupt deadline alert blob"),
+ ),
+ id="decode_failure",
+ ),
+ pytest.param(
+ datetime.timedelta(hours=1),
+ mock.patch.object(
+ SerializedReferenceModels.FixedDatetimeDeadline,
+ "evaluate_with",
+ autospec=True,
+ side_effect=RuntimeError("evaluate_with failed"),
+ ),
+ id="evaluate_with_failure",
+ ),
+ ],
+ )
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_stable(self, _, mock_get, session, deadline_test_dag):
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
+ def test_dagrun_deadline_failure_is_isolated(self, _, interval, failure, session, deadline_test_dag):
+ """A failure while creating any single deadline must not abort DagRun creation."""
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
+
+ scheduler_dag = deadline_test_dag(
+ deadline=DeadlineAlert(
+ reference=DeadlineReference.FIXED_DATETIME(future_date),
+ interval=interval,
+ callback=AsyncCallback(empty_callback_for_deadline),
+ ),
+ )
+
+ with failure:
+ dag_run = self.create_dag_run(
+ dag=scheduler_dag,
+ task_states={"task_1": TaskInstanceState.SUCCESS},
+ session=session,
+ )
- # First value used during resolution.
- mock_get.return_value = "60"
+ assert dag_run is not None
+ assert session.execute(select(Deadline)).scalars().one_or_none() is None
+
+ @mock.patch.object(Deadline, "prune_deadlines")
+ def test_dagrun_deadline_variable_interval_resolves_from_env_var(
+ self, _, session, deadline_test_dag, monkeypatch
+ ):
+ """A VariableInterval backed by an ``AIRFLOW_VAR_*`` env var (no DB row) must resolve."""
+ monkeypatch.setenv("AIRFLOW_VAR_ENV_INTERVAL_KEY", "7")
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
scheduler_dag = deadline_test_dag(
deadline=DeadlineAlert(
reference=DeadlineReference.FIXED_DATETIME(future_date),
- interval=VariableInterval("my_key"),
+ interval=VariableInterval("env_interval_key"),
callback=AsyncCallback(empty_callback_for_deadline),
),
)
dag_run = self.create_dag_run(
dag=scheduler_dag,
- task_states={"task_1": TaskInstanceState.SUCCESS, "task_2": TaskInstanceState.SUCCESS},
+ task_states={"task_1": TaskInstanceState.SUCCESS},
session=session,
)
- dag_run.dag = scheduler_dag
-
- # First update resolve interval to "5".
- dag_run.update_state(session=session)
-
- deadline = session.execute(select(Deadline)).scalars().one_or_none()
- first_deadline_time = deadline.deadline_time
-
- # Change Variable value after resolution.
- mock_get.return_value = "120"
-
- # Run again (This should not change existing deadline).
- dag_run.update_state(session=session)
+ assert dag_run is not None
deadline = session.execute(select(Deadline)).scalars().one_or_none()
- assert deadline.deadline_time == first_deadline_time
+ assert deadline is not None
+ assert deadline.deadline_time == future_date + datetime.timedelta(seconds=7)
+ @pytest.mark.parametrize(
+ ("multi_team", "team_name"),
+ [
+ pytest.param("true", "team_alpha", id="team_scoped"),
+ pytest.param("false", None, id="global"),
+ ],
+ )
@mock.patch.object(Deadline, "prune_deadlines")
- def test_dagrun_deadline_variable_interval_missing_variable_fails(self, _, session, deadline_test_dag):
- mock_err = mock.Mock()
- mock_err.error.value = "MISSING_DEADLINE"
- mock_err.detail = "missing deadline"
+ def test_dagrun_deadline_variable_interval_scoped_to_team(
+ self, _, multi_team, team_name, session, deadline_test_dag
+ ):
+ """A VariableInterval must resolve against the DagRun's team, not the global scope."""
+ future_date = datetime.datetime(2037, 1, 1, tzinfo=datetime.timezone.utc)
- with mock.patch.object(
- Variable,
- "get",
- side_effect=AirflowRuntimeError(mock_err),
- ):
- future_date = datetime.datetime.now() + datetime.timedelta(days=365)
+ scheduler_dag = deadline_test_dag(
+ deadline=DeadlineAlert(
+ reference=DeadlineReference.FIXED_DATETIME(future_date),
+ interval=VariableInterval("team_interval_key"),
+ callback=AsyncCallback(empty_callback_for_deadline),
+ ),
+ )
- scheduler_dag = deadline_test_dag(
- deadline=DeadlineAlert(
- reference=DeadlineReference.FIXED_DATETIME(future_date),
- interval=VariableInterval("missing_key"),
- callback=AsyncCallback(empty_callback_for_deadline),
- ),
+ with (
+ conf_vars({("core", "multi_team"): multi_team}),
+ mock.patch("airflow.models.dag.DagModel.get_team_name", return_value=team_name),
+ mock.patch(
+ "airflow.serialization.definitions.dag.call_secrets_backend_method",
+ return_value="5",
+ ) as mock_call,
+ ):
+ dag_run = self.create_dag_run(
+ dag=scheduler_dag,
+ task_states={"task_1": TaskInstanceState.SUCCESS},
+ session=session,
)
- with pytest.raises(ValueError, match="not found"):
- self.create_dag_run(
- dag=scheduler_dag,
- task_states={"task_1": TaskInstanceState.SUCCESS},
- session=session,
- )
+ assert dag_run is not None
+ # The team the DagRun belongs to must be forwarded to the backend lookup.
+ mock_call.assert_called_once()
+ assert mock_call.call_args.kwargs["team_name"] == team_name
Review Comment:
It's possible I'm missing something, but if the bug was that getting the Variable from the db was running into the `prohibit_commit` wrapper, none of these new tests are running into that.
The isolation half is covered, but I don't think the session-forwarding half is. If you delete just the `session=session` forwarding to `MetastoreBackend`, I believe every test here still passes: without a `prohibit_commit` guard in the test, the commit just succeeds and the Variable is still found. Could we get one test that creates the run inside the guard, the way `test_mutation_hook_committing_session_crashes_under_prohibit_commit` below does? That would actually test the thing the PR is fixing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve VariableInterval deadlines safely at DagRun creation [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68917:
URL: https://github.com/apache/airflow/pull/68917#discussion_r3694083320
##########
airflow-core/src/airflow/serialization/definitions/dag.py:
##########
@@ -741,51 +743,93 @@ def _process_dagrun_deadline_alerts(
if not deadline_alert:
continue
- deserialized_deadline_alert = decode_deadline_alert(
- {
- Encoding.TYPE: DAT.DEADLINE_ALERT,
- Encoding.VAR: {
- DeadlineAlertFields.REFERENCE: deadline_alert.reference,
- DeadlineAlertFields.INTERVAL: deadline_alert.interval,
- DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
- },
- }
- )
+ # Deadline creation is best-effort. A failure here must not prevent the DagRun
+ # itself from being created. Use a plain try/except rather than
+ # ``session.begin_nested()`` since ``create_dagrun`` runs under
+ # ``prohibit_commit`` and releasing a SAVEPOINT would trip that guard.
+ try:
+ deserialized_deadline_alert = decode_deadline_alert(
+ {
+ Encoding.TYPE: DAT.DEADLINE_ALERT,
+ Encoding.VAR: {
+ DeadlineAlertFields.REFERENCE: deadline_alert.reference,
+ DeadlineAlertFields.INTERVAL: deadline_alert.interval,
+ DeadlineAlertFields.CALLBACK: deadline_alert.callback_def,
+ },
+ }
+ )
- interval = deserialized_deadline_alert.interval
+ interval = deserialized_deadline_alert.interval
- if isinstance(interval, VariableInterval):
- interval = interval.resolve()
+ if isinstance(interval, VariableInterval):
+ interval = self._resolve_variable_interval(interval, session=session)
- if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
- deadline_time = deserialized_deadline_alert.reference.evaluate_with(
- session=session,
- interval=interval,
- # TODO : Pretty sure we can drop these last two; verify after testing is complete
- dag_id=self.dag_id,
- run_id=orm_dagrun.run_id,
- )
+ if isinstance(deserialized_deadline_alert.reference, SerializedReferenceModels.TYPES.DAGRUN):
+ deadline_time = deserialized_deadline_alert.reference.evaluate_with(
+ session=session,
+ interval=interval,
+ # TODO : Pretty sure we can drop these last two; verify after testing is complete
+ dag_id=self.dag_id,
+ run_id=orm_dagrun.run_id,
+ )
- if deadline_time is not None:
- session.add(
- Deadline(
- deadline_time=deadline_time,
- callback=deserialized_deadline_alert.callback,
- dagrun_id=orm_dagrun.id,
- deadline_alert_id=deadline_alert.id,
- dag_id=orm_dagrun.dag_id,
- bundle_name=orm_dagrun.dag_model.bundle_name,
+ if deadline_time is not None:
+ session.add(
+ Deadline(
+ deadline_time=deadline_time,
+ callback=deserialized_deadline_alert.callback,
+ dagrun_id=orm_dagrun.id,
+ deadline_alert_id=deadline_alert.id,
+ dag_id=orm_dagrun.dag_id,
+ bundle_name=orm_dagrun.dag_model.bundle_name,
+ )
)
- )
- team_name = (
- DagModel.get_team_name(self.dag_id, session=session)
- if airflow_conf.getboolean("core", "multi_team")
- else None
- )
- stats.incr(
- "deadline_alerts.deadline_created",
- tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
- )
+ team_name = (
+ DagModel.get_team_name(self.dag_id, session=session)
+ if airflow_conf.getboolean("core", "multi_team")
+ else None
+ )
+ stats.incr(
+ "deadline_alerts.deadline_created",
+ tags=prune_dict({"dag_id": self.dag_id, "team_name": team_name}),
+ )
+ except Exception:
+ log.exception(
+ "Failed to create deadline for alert %s on DagRun %s (dag_id=%s); "
+ "skipping this deadline, the DagRun is unaffected",
+ getattr(deadline_alert, "id", "<unknown>"),
+ orm_dagrun.run_id,
+ self.dag_id,
+ )
+ stats.incr("deadline_alerts.deadline_creation_failed", tags={"dag_id": self.dag_id})
+
+ @staticmethod
+ def _resolve_variable_interval(interval: VariableInterval, *, session: Session) -> datetime.timedelta:
+ """
+ Resolve a ``VariableInterval`` to a concrete ``timedelta`` at DagRun creation.
+
+ The Variable is resolved using the standard secrets lookup order. The scheduler
+ session is passed to the metastore backend to avoid creating a new session
+ during DagRun creation.
+
+ :param interval: The ``VariableInterval`` to resolve.
+ :param session: Scheduler session used for metadata database lookups.
+ :return: The resolved ``timedelta``.
+ :raises ValueError: If the Variable cannot be resolved or converted to a valid ``timedelta``.
+ """
+ for backend in ensure_secrets_loaded():
+ value = call_secrets_backend_method(
+ backend.get_variable,
+ team_name=None,
Review Comment:
Should `team_name` be hardcoded to `None` here?? Seems like it should check if there is a team unless I;'m missing something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not abort Dag run creation when a deadline alert fails [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi merged PR #68917:
URL: https://github.com/apache/airflow/pull/68917
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org