You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "seanghaeli (via GitHub)" <gi...@apache.org> on 2026/05/04 21:55:41 UTC

[PR] Add remote log upload support for callback subprocesses [airflow]

seanghaeli opened a new pull request, #66379:
URL: https://github.com/apache/airflow/pull/66379

   ## Summary
   
   Enable callback logs (deadline alerts, SLA notifications, etc.) to be uploaded to remote storage (S3, GCS, etc.) so they are visible in the Airflow UI when remote logging is configured.
   
   Key change: make `ti` optional in `RemoteLogIO.upload()` protocol and `upload_to_remote()` since callbacks don't have a task instance. The 7 providers that ignore `ti` (S3, GCS, CloudWatch, WASB, OSS, HDFS, Alibaba) work immediately. Elasticsearch and OpenSearch use `ti` for log ID rendering and will need a follow-up to handle the `ti=None` case — until then, callback log upload failures for those providers are caught and logged without affecting callback execution.
   
   ## Changes
   
   - Make `ti` optional (`RuntimeTI | None = None`) in `RemoteLogIO.upload()` protocol and `upload_to_remote()`
   - Add `_upload_logs()` to `CallbackSubprocess.wait()`, delegating to `upload_to_remote` with error handling that swallows failures
   - Wire up `_remote_logging_conn(client)` in `_configure_logging()` so remote log handler processors are initialized during logging setup
   
   ## Related
   
   - Follows #65269 (callback supervisor comms handlers) which added the Connection/Variable access infrastructure this builds on
   - Prerequisite for callback log visibility in the UI (tracked on Asana Deadlines board)
   
   ^ Add meaningful description above
   
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst) for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html).
   In case of backwards incompatible changes please leave a note in a newsfragment/[PR_NUMBER].significant.rst file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on PR #66379:
URL: https://github.com/apache/airflow/pull/66379#issuecomment-4566203747

   Addressed all review feedback:
   - Ferruzzi nit: removed the redundant docstring text from `wait()`
   - Ramitkataria: changed `id=self.id` → `callback_id=self.id` in the log field
   - Ramitkataria: `test_wait_calls_upload_logs_after_subprocess_completes` already covers that `wait()` calls `_upload_logs()` after subprocess exits
   - Eladkal: removed the warning (resolved in earlier commit)
   
   All threads now have replies. CI is running on the latest commit.
   
   ---
   Drafted-by: Claude Code (claude-sonnet-4-6); reviewed by @seanghaeli before posting


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on PR #66379:
URL: https://github.com/apache/airflow/pull/66379#issuecomment-4410217646

   @ferruzzi @ramitkataria your reviews would be much appreciated


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #66379:
URL: https://github.com/apache/airflow/pull/66379#discussion_r3230352029


##########
task-sdk/src/airflow/sdk/execution_time/callback_supervisor.py:
##########
@@ -242,8 +242,23 @@ def wait(self) -> int:
             self.selector.close()
 
         self._exit_code = self._exit_code if self._exit_code is not None else 1
+
+        # Upload logs to remote storage for UI visibility (same pattern as ActivitySubprocess)

Review Comment:
   Nittiest of nitpicks, can likely drop this comment.  The method name is pretty clear IMHO.  Or just resolve this, no real opinion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #66379:
URL: https://github.com/apache/airflow/pull/66379#discussion_r3319295744


##########
task-sdk/src/airflow/sdk/execution_time/callback_supervisor.py:
##########
@@ -242,8 +242,23 @@ def wait(self) -> int:
             self.selector.close()
 
         self._exit_code = self._exit_code if self._exit_code is not None else 1
+
+        # Upload logs to remote storage for UI visibility (same pattern as ActivitySubprocess)

Review Comment:
   Done — removed the redundant text from the docstring in the latest commit.



##########
task-sdk/src/airflow/sdk/execution_time/callback_supervisor.py:
##########
@@ -244,8 +244,22 @@ def wait(self) -> int:
             self.selector.close()
 
         self._exit_code = self._exit_code if self._exit_code is not None else 1
+
+        self._upload_logs()
+
         return self._exit_code
 
+    def _upload_logs(self):
+        """Upload callback logs to remote storage if configured."""
+        from airflow.sdk.execution_time.supervisor import _remote_logging_conn
+        from airflow.sdk.log import upload_to_remote
+
+        try:
+            with _remote_logging_conn(self.client):
+                upload_to_remote(self.process_log)
+        except Exception:
+            log.exception("Failed to upload callback logs to remote storage")

Review Comment:
   Fixed — now uses callback_id= to match the pattern in ActivitySubprocess.



##########
task-sdk/tests/task_sdk/execution_time/test_callback_supervisor.py:
##########
@@ -239,3 +239,70 @@ def test_handle_requests(
 
         if client_mock:
             mock_client_method.assert_called_once_with(*client_mock.args, **client_mock.kwargs)
+
+
+class TestConfigureLogging:
+    """Tests for _configure_logging remote logging connection setup."""
+
+    def test_configure_logging_uses_remote_logging_conn(self, tmp_path, mocker):
+        """Verify that _remote_logging_conn is invoked with the client during logging setup."""
+        from airflow.sdk.execution_time.callback_supervisor import _configure_logging
+
+        mock_client = mocker.Mock()
+        log_path = str(tmp_path / "callback.log")
+
+        mock_remote_conn = mocker.patch(
+            "airflow.sdk.execution_time.supervisor._remote_logging_conn",
+        )
+
+        logger, fd = _configure_logging(log_path, mock_client)
+        fd.close()
+
+        mock_remote_conn.assert_called_once_with(mock_client)
+
+
+class TestUploadLogs:

Review Comment:
   The test test_wait_calls_upload_logs_after_subprocess_completes already covers this — it patches _upload_logs and asserts it was called after wait() returns.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on PR #66379:
URL: https://github.com/apache/airflow/pull/66379#issuecomment-4568034040

   I think it LGTM; my concerns are addressed.   @ramitkataria ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #66379:
URL: https://github.com/apache/airflow/pull/66379#issuecomment-4535926066

   @amoghrajesh could you take a look please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on code in PR #66379:
URL: https://github.com/apache/airflow/pull/66379#discussion_r3345105085


##########
providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py:
##########
@@ -691,8 +691,11 @@ def _get_source_includes(self, extra_fields: Iterable[str] = ()) -> list[str]:
             ["@timestamp", *TASK_LOG_FIELDS, self.host_field, self.offset_field, *extra_fields]
         )
 
-    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
         """Write the log to ElasticSearch."""
+        if ti is None:

Review Comment:
   Should we have a unit test for this path?



##########
providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py:
##########
@@ -853,8 +853,11 @@ def __attrs_post_init__(self):
         self._doc_type_map: dict[Any, Any] = {}
         self._doc_type: list[Any] = []
 
-    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
         """Emit structured task logs to stdout and/or write them directly to OpenSearch."""
+        if ti is None:

Review Comment:
   Same here - a test for this branch would be good to have



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on code in PR #66379:
URL: https://github.com/apache/airflow/pull/66379#discussion_r3284948214


##########
task-sdk/src/airflow/sdk/execution_time/callback_supervisor.py:
##########
@@ -244,8 +244,22 @@ def wait(self) -> int:
             self.selector.close()
 
         self._exit_code = self._exit_code if self._exit_code is not None else 1
+
+        self._upload_logs()
+
         return self._exit_code
 
+    def _upload_logs(self):
+        """Upload callback logs to remote storage if configured."""
+        from airflow.sdk.execution_time.supervisor import _remote_logging_conn
+        from airflow.sdk.log import upload_to_remote
+
+        try:
+            with _remote_logging_conn(self.client):
+                upload_to_remote(self.process_log)
+        except Exception:
+            log.exception("Failed to upload callback logs to remote storage")

Review Comment:
   Nit: the `ActivitySubprocess` equivalent at `supervisor.py:1381` logs structured context for debuggability:
   `self.process_log.exception("Failed to upload remote logs", ti_id=self.id, pid=self.pid)`
   So, it would be nice to include similar fields here. When multiple callbacks run concurrently, the bare message won't tell you which one failed. Maybe something like:
   
   ```suggestion
               log.exception("Failed to upload callback logs to remote storage", callback_id=self.id, pid=self.pid)
   ```



##########
task-sdk/tests/task_sdk/execution_time/test_callback_supervisor.py:
##########
@@ -239,3 +239,70 @@ def test_handle_requests(
 
         if client_mock:
             mock_client_method.assert_called_once_with(*client_mock.args, **client_mock.kwargs)
+
+
+class TestConfigureLogging:
+    """Tests for _configure_logging remote logging connection setup."""
+
+    def test_configure_logging_uses_remote_logging_conn(self, tmp_path, mocker):
+        """Verify that _remote_logging_conn is invoked with the client during logging setup."""
+        from airflow.sdk.execution_time.callback_supervisor import _configure_logging
+
+        mock_client = mocker.Mock()
+        log_path = str(tmp_path / "callback.log")
+
+        mock_remote_conn = mocker.patch(
+            "airflow.sdk.execution_time.supervisor._remote_logging_conn",
+        )
+
+        logger, fd = _configure_logging(log_path, mock_client)
+        fd.close()
+
+        mock_remote_conn.assert_called_once_with(mock_client)
+
+
+class TestUploadLogs:

Review Comment:
   Nit: these test `_upload_logs()` directly, which is great for validating its internal behaviour but it would be nice to have a small test that verifies wait() actually calls _upload_logs() after the subprocess exits



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on code in PR #66379:
URL: https://github.com/apache/airflow/pull/66379#discussion_r3290677279


##########
providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py:
##########
@@ -666,8 +666,14 @@ def _get_source_includes(self, extra_fields: Iterable[str] = ()) -> list[str]:
             ["@timestamp", *TASK_LOG_FIELDS, self.host_field, self.offset_field, *extra_fields]
         )
 
-    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
         """Write the log to ElasticSearch."""
+        if ti is None:
+            self.log.warning(
+                "Skipping Elasticsearch log upload: task instance is required for log ID rendering"

Review Comment:
   Agreed, removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli commented on PR #66379:
URL: https://github.com/apache/airflow/pull/66379#issuecomment-4614936900

   Let's widen the signatures in a follow-up if we want strict protocol conformance, but it felt like unnecessary churn here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66379:
URL: https://github.com/apache/airflow/pull/66379#issuecomment-4476938940

   @seanghaeli — There are 1 unresolved review thread on this PR from @ferruzzi. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? Once you believe the feedback is addressed, mark the thread as resolved so the reviewer isn't re-pinged needlessly. Thanks!
   
   ---
   
   _Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "seanghaeli (via GitHub)" <gi...@apache.org>.
seanghaeli closed pull request #66379: Add remote log upload support for callback subprocesses
URL: https://github.com/apache/airflow/pull/66379


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #66379:
URL: https://github.com/apache/airflow/pull/66379#discussion_r3290563684


##########
providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py:
##########
@@ -666,8 +666,14 @@ def _get_source_includes(self, extra_fields: Iterable[str] = ()) -> list[str]:
             ["@timestamp", *TASK_LOG_FIELDS, self.host_field, self.offset_field, *extra_fields]
         )
 
-    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
         """Write the log to ElasticSearch."""
+        if ti is None:
+            self.log.warning(
+                "Skipping Elasticsearch log upload: task instance is required for log ID rendering"

Review Comment:
   I am not sure users will understand this warning.
   I mean, when user see this - what is he expected to do?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add remote log upload support for callback subprocesses [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #66379:
URL: https://github.com/apache/airflow/pull/66379


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org