You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ephraimbuddy (via GitHub)" <gi...@apache.org> on 2026/04/20 14:43:43 UTC
[PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
ephraimbuddy opened a new pull request, #65543:
URL: https://github.com/apache/airflow/pull/65543
Split the DagFileProcessorManager callback path into discrete methods so metadata-DB access can be swapped for API calls:
- fetch_callbacks() — the outer seam; default hits the DB
- resolve_callback_bundle() — bundle lookup
- initialize_callback_bundle() — bundle preparation, returning whether the callback can proceed
No behavior change for the default DB-backed flow.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes (please specify the tool below)
Generated-by: claude opus 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3152266853
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -568,13 +568,21 @@ def _claim_priority_files(self, session: Session = NEW_SESSION) -> list[DagFileI
session.delete(request)
return files
+ def fetch_callbacks(self) -> list[CallbackRequest]:
+ """
+ Fetch and claim callbacks for this manager's bundles.
+
+ Default implementation reads from the metadata DB; override to source callbacks from an API.
+ """
+ return self._fetch_callbacks_from_db()
+
@provide_session
@retry_db_transaction
- def _fetch_callbacks(
+ def _fetch_callbacks_from_db(
self,
session: Session = NEW_SESSION,
) -> list[CallbackRequest]:
- """Fetch callbacks from database and add them to the internal queue for execution."""
+ """Fetch callbacks from the metadata database."""
Review Comment:
```suggestion
"""Fetch callbacks from database and add them to the internal queue for execution."""
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3117844268
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,24 +589,46 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Raises :class:`CallbackBundleUnavailable` if the bundle cannot be resolved.
+ Override to source the bundle from an API.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
- except ValueError:
- # Bundle no longer configured
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ except ValueError as e:
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
- return None
+ raise CallbackBundleUnavailable(f"Bundle {request.bundle_name!r} no longer configured") from e
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> None:
+ """
+ Prepare ``bundle`` so the callback can run against it.
+
+ Raises :class:`CallbackBundleUnavailable` if preparation fails and the callback should
+ be skipped. Override to change how bundles are materialized (for example, via an API).
+ """
Review Comment:
PR description says `initialize_callback_bundle()` returns whether the callback can proceed, but the implementation returns `None` and uses `CallbackBundleUnavailable` exceptions as the control-flow signal. Please either update the PR description (and/or method docstring) to reflect the exception-based contract, or change the method to match the documented return-value behavior so override authors aren’t misled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3118013019
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,24 +589,46 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Raises :class:`CallbackBundleUnavailable` if the bundle cannot be resolved.
+ Override to source the bundle from an API.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
- except ValueError:
- # Bundle no longer configured
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ except ValueError as e:
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
- return None
+ raise CallbackBundleUnavailable(f"Bundle {request.bundle_name!r} no longer configured") from e
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> None:
+ """
+ Prepare ``bundle`` so the callback can run against it.
+
+ Raises :class:`CallbackBundleUnavailable` if preparation fails and the callback should
+ be skipped. Override to change how bundles are materialized (for example, via an API).
+ """
if bundle.supports_versioning and request.bundle_version:
try:
bundle.initialize()
- except Exception:
+ except Exception as e:
self.log.exception(
"Error initializing bundle %s version %s for callback, skipping",
request.bundle_name,
request.bundle_version,
)
- return None
+ raise CallbackBundleUnavailable(
Review Comment:
Where are these exceptions caught?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3119153745
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,24 +589,46 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Raises :class:`CallbackBundleUnavailable` if the bundle cannot be resolved.
+ Override to source the bundle from an API.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
- except ValueError:
- # Bundle no longer configured
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ except ValueError as e:
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
- return None
+ raise CallbackBundleUnavailable(f"Bundle {request.bundle_name!r} no longer configured") from e
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> None:
+ """
+ Prepare ``bundle`` so the callback can run against it.
+
+ Raises :class:`CallbackBundleUnavailable` if preparation fails and the callback should
+ be skipped. Override to change how bundles are materialized (for example, via an API).
+ """
if bundle.supports_versioning and request.bundle_version:
try:
bundle.initialize()
- except Exception:
+ except Exception as e:
Review Comment:
Removed the exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3146590268
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -535,13 +535,21 @@ def _get_priority_files(self, session: Session = NEW_SESSION) -> list[DagFileInf
session.delete(request)
return files
+ def fetch_callbacks(self) -> list[CallbackRequest]:
+ """
+ Fetch and claim callbacks for this manager's bundles.
+
+ Default implementation reads from the metadata DB; override to source callbacks from an API.
+ """
+ return self._fetch_callbacks()
+
@provide_session
@retry_db_transaction
def _fetch_callbacks(
Review Comment:
Maybe rename this to `_fetch_callbacks_from_db`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3112355970
##########
airflow-core/tests/unit/dag_processing/test_manager.py:
##########
@@ -1543,6 +1543,80 @@ def test_callback_queue(self, mock_get_logger, configure_testing_dag_bundle):
assert dag1_path not in manager._callback_to_execute
assert dag2_path not in manager._callback_to_execute
+ @mock.patch("airflow.dag_processing.manager.DagBundlesManager")
+ def test_resolve_callback_bundle(self, mock_bundle_manager):
+ manager = DagFileProcessorManager(max_runs=1)
+ bundle = MagicMock()
+ mock_bundle_manager.return_value.get_bundle.return_value = bundle
+
+ request = DagCallbackRequest(
+ filepath="file1.py",
+ dag_id="dag1",
+ run_id="run1",
+ is_failure_callback=False,
+ bundle_name="testing",
+ bundle_version="some_commit_hash",
+ msg=None,
+ )
+
+ assert manager.resolve_callback_bundle(request) is bundle
+ mock_bundle_manager.return_value.get_bundle.assert_called_once_with(
+ name="testing", version="some_commit_hash"
+ )
+
+ @mock.patch("airflow.dag_processing.manager.DagBundlesManager")
+ def test_resolve_callback_bundle_returns_none_for_unconfigured_bundle(self, mock_bundle_manager):
+ manager = DagFileProcessorManager(max_runs=1)
+ mock_bundle_manager.return_value.get_bundle.side_effect = ValueError("missing bundle")
+
+ request = DagCallbackRequest(
+ filepath="file1.py",
+ dag_id="dag1",
+ run_id="run1",
+ is_failure_callback=False,
+ bundle_name="testing",
+ bundle_version="some_commit_hash",
+ msg=None,
+ )
+
+ assert manager.resolve_callback_bundle(request) is None
+
+ def test_initialize_callback_bundle_skips_unversioned_callback(self):
+ manager = DagFileProcessorManager(max_runs=1)
+ bundle = MagicMock()
+ bundle.supports_versioning = True
+
+ request = DagCallbackRequest(
+ filepath="file1.py",
+ dag_id="dag1",
+ run_id="run1",
+ is_failure_callback=False,
+ bundle_name="testing",
+ bundle_version=None,
+ msg=None,
+ )
+
+ assert manager.initialize_callback_bundle(request, bundle) is True
+ bundle.initialize.assert_not_called()
+
+ def test_initialize_callback_bundle_skips_non_versioned_bundle(self):
+ manager = DagFileProcessorManager(max_runs=1)
+ bundle = MagicMock()
+ bundle.supports_versioning = False
Review Comment:
New tests create `MagicMock()` instances without `spec`/`autospec`, which can hide attribute errors and make tests less meaningful. Prefer `MagicMock(spec=...)` so the mock reflects the real bundle interface used by `initialize_callback_bundle`.
##########
airflow-core/tests/unit/dag_processing/test_manager.py:
##########
@@ -1543,6 +1543,80 @@ def test_callback_queue(self, mock_get_logger, configure_testing_dag_bundle):
assert dag1_path not in manager._callback_to_execute
assert dag2_path not in manager._callback_to_execute
+ @mock.patch("airflow.dag_processing.manager.DagBundlesManager")
+ def test_resolve_callback_bundle(self, mock_bundle_manager):
+ manager = DagFileProcessorManager(max_runs=1)
+ bundle = MagicMock()
+ mock_bundle_manager.return_value.get_bundle.return_value = bundle
Review Comment:
New tests create `MagicMock()` instances without `spec`/`autospec`, which can hide attribute errors and make tests less meaningful. Prefer `MagicMock(spec=...)` (or use `autospec=True` on patch) so the mocks match the real bundle interface.
##########
airflow-core/tests/unit/dag_processing/test_manager.py:
##########
@@ -1543,6 +1543,80 @@ def test_callback_queue(self, mock_get_logger, configure_testing_dag_bundle):
assert dag1_path not in manager._callback_to_execute
assert dag2_path not in manager._callback_to_execute
+ @mock.patch("airflow.dag_processing.manager.DagBundlesManager")
+ def test_resolve_callback_bundle(self, mock_bundle_manager):
+ manager = DagFileProcessorManager(max_runs=1)
+ bundle = MagicMock()
+ mock_bundle_manager.return_value.get_bundle.return_value = bundle
+
+ request = DagCallbackRequest(
+ filepath="file1.py",
+ dag_id="dag1",
+ run_id="run1",
+ is_failure_callback=False,
+ bundle_name="testing",
+ bundle_version="some_commit_hash",
+ msg=None,
+ )
+
+ assert manager.resolve_callback_bundle(request) is bundle
+ mock_bundle_manager.return_value.get_bundle.assert_called_once_with(
+ name="testing", version="some_commit_hash"
+ )
+
+ @mock.patch("airflow.dag_processing.manager.DagBundlesManager")
+ def test_resolve_callback_bundle_returns_none_for_unconfigured_bundle(self, mock_bundle_manager):
+ manager = DagFileProcessorManager(max_runs=1)
+ mock_bundle_manager.return_value.get_bundle.side_effect = ValueError("missing bundle")
+
+ request = DagCallbackRequest(
+ filepath="file1.py",
+ dag_id="dag1",
+ run_id="run1",
+ is_failure_callback=False,
+ bundle_name="testing",
+ bundle_version="some_commit_hash",
+ msg=None,
+ )
+
+ assert manager.resolve_callback_bundle(request) is None
+
+ def test_initialize_callback_bundle_skips_unversioned_callback(self):
+ manager = DagFileProcessorManager(max_runs=1)
+ bundle = MagicMock()
+ bundle.supports_versioning = True
Review Comment:
New tests create `MagicMock()` instances without `spec`/`autospec`, which can hide attribute errors and make tests less meaningful. Prefer `MagicMock(spec=...)` so the mock matches `BaseDagBundle` (or the concrete bundle type) and fails fast on invalid attribute access.
##########
airflow-core/tests/unit/dag_processing/test_manager.py:
##########
@@ -1565,6 +1639,25 @@ def test_add_callback_initializes_versioned_bundle(self, mock_bundle_manager):
bundle.initialize.assert_called_once()
+ def test_initialize_callback_bundle_returns_false_when_initialization_fails(self):
+ manager = DagFileProcessorManager(max_runs=1)
+ bundle = MagicMock()
+ bundle.supports_versioning = True
+ bundle.initialize.side_effect = Exception("clone failed")
+
Review Comment:
New tests create `MagicMock()` instances without `spec`/`autospec`, which can hide attribute errors and make tests less meaningful. Prefer `MagicMock(spec=...)` (or `create_autospec`) so `initialize_callback_bundle` interactions are validated against the real bundle API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3116370142
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,14 +577,26 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle | None:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Returns ``None`` if the bundle is no longer configured.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
except ValueError:
- # Bundle no longer configured
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
return None
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> bool:
Review Comment:
I think keeping it as bool creates a clearer seam for overrides. With bool, an override's contract is: log your own specifics, return False to skip. With exceptions, overrides has to understand that the caller will catch exception and log the db error "error initializing bundle" regardless of what really failed e.g http errors from an API-backed override). Main goal is to be able to swap the db calls with api backed calls. Also, resolve_callback_bundle already returns None for the same reason rather than raising
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3119142519
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,24 +589,46 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Raises :class:`CallbackBundleUnavailable` if the bundle cannot be resolved.
+ Override to source the bundle from an API.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
- except ValueError:
- # Bundle no longer configured
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ except ValueError as e:
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
- return None
+ raise CallbackBundleUnavailable(f"Bundle {request.bundle_name!r} no longer configured") from e
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> None:
+ """
+ Prepare ``bundle`` so the callback can run against it.
+
+ Raises :class:`CallbackBundleUnavailable` if preparation fails and the callback should
+ be skipped. Override to change how bundles are materialized (for example, via an API).
+ """
if bundle.supports_versioning and request.bundle_version:
try:
bundle.initialize()
- except Exception:
+ except Exception as e:
self.log.exception(
"Error initializing bundle %s version %s for callback, skipping",
request.bundle_name,
request.bundle_version,
)
- return None
+ raise CallbackBundleUnavailable(
+ f"Failed to initialize bundle {request.bundle_name!r} version {request.bundle_version!r}"
+ ) from e
+
+ def _add_callback_to_queue(self, request: CallbackRequest) -> None:
+ self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ try:
+ bundle = self.resolve_callback_bundle(request)
+ self.initialize_callback_bundle(request, bundle)
Review Comment:
Thanks. I have collapsed them into `prepare_callback_bundle`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3118690521
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -535,13 +547,21 @@ def _get_priority_files(self, session: Session = NEW_SESSION) -> list[DagFileInf
session.delete(request)
return files
+ def fetch_callbacks(self) -> list[CallbackRequest]:
Review Comment:
`fetch_callbacks` isn't covered directly. A revert to `self._fetch_callbacks()` on line 462 would still pass the existing tests since the default just delegates. Worth a one-liner test that asserts `fetch_callbacks()` returns what `_fetch_callbacks()` returns, so the seam can't regress silently.
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,24 +589,46 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Raises :class:`CallbackBundleUnavailable` if the bundle cannot be resolved.
+ Override to source the bundle from an API.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
- except ValueError:
- # Bundle no longer configured
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ except ValueError as e:
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
- return None
+ raise CallbackBundleUnavailable(f"Bundle {request.bundle_name!r} no longer configured") from e
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> None:
+ """
+ Prepare ``bundle`` so the callback can run against it.
+
+ Raises :class:`CallbackBundleUnavailable` if preparation fails and the callback should
+ be skipped. Override to change how bundles are materialized (for example, via an API).
+ """
if bundle.supports_versioning and request.bundle_version:
try:
bundle.initialize()
- except Exception:
+ except Exception as e:
Review Comment:
If a subclass override calls back into `super().initialize_callback_bundle(...)` or chains to another helper that raises `CallbackBundleUnavailable`, this `except Exception` catches it and re-wraps with the generic "Failed to initialize bundle..." message, losing the original reason. Consider narrowing to the expected exception types or letting `CallbackBundleUnavailable` pass through.
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,24 +589,46 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Raises :class:`CallbackBundleUnavailable` if the bundle cannot be resolved.
+ Override to source the bundle from an API.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
- except ValueError:
- # Bundle no longer configured
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ except ValueError as e:
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
- return None
+ raise CallbackBundleUnavailable(f"Bundle {request.bundle_name!r} no longer configured") from e
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> None:
+ """
+ Prepare ``bundle`` so the callback can run against it.
+
+ Raises :class:`CallbackBundleUnavailable` if preparation fails and the callback should
+ be skipped. Override to change how bundles are materialized (for example, via an API).
+ """
Review Comment:
The docstring says "prepare bundle so the callback can run" but when `supports_versioning` is False or `bundle_version` is None, this silently no-ops. That's deliberate for the DB-backed flow, but an override author reading only the docstring may not realize they also inherit the skip-when-unversioned behavior. Could be explicit: "Default implementation is a no-op for unversioned bundles or requests without a version."
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,24 +589,46 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Raises :class:`CallbackBundleUnavailable` if the bundle cannot be resolved.
+ Override to source the bundle from an API.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
- except ValueError:
- # Bundle no longer configured
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ except ValueError as e:
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
- return None
+ raise CallbackBundleUnavailable(f"Bundle {request.bundle_name!r} no longer configured") from e
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> None:
+ """
+ Prepare ``bundle`` so the callback can run against it.
+
+ Raises :class:`CallbackBundleUnavailable` if preparation fails and the callback should
+ be skipped. Override to change how bundles are materialized (for example, via an API).
+ """
if bundle.supports_versioning and request.bundle_version:
try:
bundle.initialize()
- except Exception:
+ except Exception as e:
self.log.exception(
"Error initializing bundle %s version %s for callback, skipping",
request.bundle_name,
request.bundle_version,
)
- return None
+ raise CallbackBundleUnavailable(
+ f"Failed to initialize bundle {request.bundle_name!r} version {request.bundle_version!r}"
+ ) from e
+
+ def _add_callback_to_queue(self, request: CallbackRequest) -> None:
+ self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ try:
+ bundle = self.resolve_callback_bundle(request)
+ self.initialize_callback_bundle(request, bundle)
+ except CallbackBundleUnavailable:
Review Comment:
The swallow here relies on raise sites logging before raising, which both default impls do. If an API-backed override raises `CallbackBundleUnavailable` without logging, the callback will be silently dropped. A `self.log.debug("Skipping callback %s: %s", request, exc)` here would make the contract self-enforcing.
##########
airflow-core/tests/unit/dag_processing/test_manager.py:
##########
@@ -1565,10 +1641,30 @@ def test_add_callback_initializes_versioned_bundle(self, mock_bundle_manager):
bundle.initialize.assert_called_once()
Review Comment:
This asserts `bundle.initialize` was called but not that `_callback_to_execute` was actually populated. Pre-existing gap, but now asymmetric with the new skip-case tests below which do assert the negative. Adding `assert manager._callback_to_execute` here would balance the coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3116426251
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,14 +577,26 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle | None:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Returns ``None`` if the bundle is no longer configured.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
except ValueError:
- # Bundle no longer configured
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
return None
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> bool:
Review Comment:
What bugs me is that the semantic version is not right. I usually won't expect a function name `initialize_callback_bundle` to return `bool`.
`resolve_callback_bundle` looks more like a silent failure/error to me. not as counter-intuitive as `initialize_callback_.bundle.`
> One alternative that would address the consistency angle is going exception-based for both resolve_callback_bundle and this one with a dedicated CallbackBundleUnavailable exception type, so the caller can narrow the catch
Yep, this sounds like a good idea!
---
how often do you think this would happen? if not that often, i would prefer exception based.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3119150414
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -535,13 +547,21 @@ def _get_priority_files(self, session: Session = NEW_SESSION) -> list[DagFileInf
session.delete(request)
return files
+ def fetch_callbacks(self) -> list[CallbackRequest]:
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3118024544
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,24 +589,46 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Raises :class:`CallbackBundleUnavailable` if the bundle cannot be resolved.
+ Override to source the bundle from an API.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
- except ValueError:
- # Bundle no longer configured
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ except ValueError as e:
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
- return None
+ raise CallbackBundleUnavailable(f"Bundle {request.bundle_name!r} no longer configured") from e
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> None:
+ """
+ Prepare ``bundle`` so the callback can run against it.
+
+ Raises :class:`CallbackBundleUnavailable` if preparation fails and the callback should
+ be skipped. Override to change how bundles are materialized (for example, via an API).
+ """
if bundle.supports_versioning and request.bundle_version:
try:
bundle.initialize()
- except Exception:
+ except Exception as e:
self.log.exception(
"Error initializing bundle %s version %s for callback, skipping",
request.bundle_name,
request.bundle_version,
)
- return None
+ raise CallbackBundleUnavailable(
+ f"Failed to initialize bundle {request.bundle_name!r} version {request.bundle_version!r}"
+ ) from e
+
+ def _add_callback_to_queue(self, request: CallbackRequest) -> None:
+ self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ try:
+ bundle = self.resolve_callback_bundle(request)
+ self.initialize_callback_bundle(request, bundle)
Review Comment:
Oh caught here.
Do these need to be separate functions? It's not immediately clear to me what the difference between resolve and initialize do differently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3115853029
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,14 +577,26 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle | None:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Returns ``None`` if the bundle is no longer configured.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
except ValueError:
- # Bundle no longer configured
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
return None
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> bool:
Review Comment:
I feel we should use error handling instead of returning a `bool` (i.e., raise an exception and handle it in _add_callback_to_queue)
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -586,7 +606,16 @@ def _add_callback_to_queue(self, request: CallbackRequest):
request.bundle_name,
request.bundle_version,
)
- return None
+ return False
+ return True
+
+ def _add_callback_to_queue(self, request: CallbackRequest):
+ self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ bundle = self.resolve_callback_bundle(request)
+ if bundle is None:
+ return None
+ if not self.initialize_callback_bundle(request, bundle):
+ return None
Review Comment:
```suggestion
def _add_callback_to_queue(self, request: CallbackRequest) -> None:
self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
bundle = self.resolve_callback_bundle(request)
if bundle is None:
return
try:
self.initialize_callback_bundle(request, bundle):
except Exception:
return
```
I guess we won't be able to narrow the exception here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3116387727
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -569,14 +577,26 @@ def _fetch_callbacks(
guard.commit()
return callback_queue
- def _add_callback_to_queue(self, request: CallbackRequest):
- self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ def resolve_callback_bundle(self, request: CallbackRequest) -> BaseDagBundle | None:
+ """
+ Resolve the bundle required to execute a callback request.
+
+ Returns ``None`` if the bundle is no longer configured.
+ """
try:
- bundle = DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
+ return DagBundlesManager().get_bundle(name=request.bundle_name, version=request.bundle_version)
except ValueError:
- # Bundle no longer configured
self.log.error("Bundle %s no longer configured, skipping callback", request.bundle_name)
return None
+
+ def initialize_callback_bundle(self, request: CallbackRequest, bundle: BaseDagBundle) -> bool:
Review Comment:
Happy to revisit if you feel strongly. One alternative that would address the consistency angle is going exception-based for both resolve_callback_bundle and this one with a dedicated CallbackBundleUnavailable exception type, so the caller can narrow the catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #65543:
URL: https://github.com/apache/airflow/pull/65543#discussion_r3116416355
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -586,7 +606,16 @@ def _add_callback_to_queue(self, request: CallbackRequest):
request.bundle_name,
request.bundle_version,
)
- return None
+ return False
+ return True
+
+ def _add_callback_to_queue(self, request: CallbackRequest):
+ self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
+ bundle = self.resolve_callback_bundle(request)
+ if bundle is None:
+ return None
+ if not self.initialize_callback_bundle(request, bundle):
+ return None
Review Comment:
Yeah, part of why I prefer we leave it with bool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Extract seams for Dag processor callback fetching to allow DB-to-API swap [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W merged PR #65543:
URL: https://github.com/apache/airflow/pull/65543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org