You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "korex-f (via GitHub)" <gi...@apache.org> on 2026/05/09 19:26:42 UTC

[PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

korex-f opened a new pull request, #66633:
URL: https://github.com/apache/airflow/pull/66633

   Three related bugs all produce `WatchtowerWarning: "Received message after logging system shutdown"`, causing CloudWatch log streams to be truncated or never created.
   
   **Bug 1** (`task-sdk/src/airflow/sdk/log.py`):
   In `configure_logging()`, `remote.processors` was accessed before the inner `configure_logging()` call ran `dictConfig()`. `dictConfig()` calls `_clearExistingHandlers()` → `logging.shutdown()` on all existing handlers, killing the just-created watchtower handler (`shutting_down=True`) before any task log is emitted. Fix: move `getattr(remote, "processors")` to after `dictConfig` runs, injecting via a second `structlog.configure()` call.
   
   **Bug 2** (`providers/amazon/.../cloudwatch_task_handler.py`):
   `CloudWatchRemoteLogIO.handler` was a `cached_property` — once killed by `logging.shutdown()` the dead instance was never replaced. The `processors` `cached_property` also captured `_handler` in a closure, pinning the dead instance. Fix: convert `handler` to a regular property that recreates when `shutting_down=True`; access `self.handler` dynamically in the `processors` closure.
   
   **Bug 3** (`task-sdk/src/airflow/sdk/execution_time/supervisor.py`):
   `_configure_logging()` returned a plain tuple with no cleanup path. The only code that ever closed the remote handler was `logging.shutdown()` at process exit, which fired after `supervise_task()` returned while final task messages were still in flight. Fix: convert `_configure_logging()` to a `contextmanager` that explicitly flushes and closes the handler after `process.wait()` drains all task log messages.
   
   closes: #66475


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5043819996

   > Picking up on @o-nikolas's merge question -- I think this needs a second look at scope before it goes in, because a fair bit of it landed via other PRs while this was open:
   > 
   >     * **[Fix CloudWatch remote logging for ephemeral lifecycle executor #68779](https://github.com/apache/airflow/pull/68779)** (merged) made the CloudWatch `RemoteLogIO` self-healing: the `handler` property rebuilds if `dictConfig` closed it mid-task, and `upload()` now calls `close()` to flush pending events. That covers the [Cloudwatch remote logging does not work for ECS Executor #66475](https://github.com/apache/airflow/issues/66475) symptom at the provider layer.
   > 
   >     * **[Upload task logs even when final state update fails #67935](https://github.com/apache/airflow/pull/67935)** (merged) moved `_upload_logs()` into a `finally` after `update_task_state_if_needed()`, so logs upload even when the state update raises.
   > 
   >     * **[Remote logging fix #68370](https://github.com/apache/airflow/pull/68370)** (merged) fixed the cached-`None`-connection case so the worker can upload at all.
   > 
   > 
   > Given that, the supervisor.py side here (explicit `_close_remote_log_handler` on teardown) now overlaps with `_upload_logs()` → `upload()` → `close()`, which already runs on the state-update error path after #67935. The log.py reorder (build the handler after `dictConfig`) is the one piece not on main, but with the self-healing handler it's no longer a live drop -- more of a cleaner-root-cause argument.
   > 
   > @korex-f -- is there a failure mode left that the three merged PRs don't cover? If the log.py reorder is the remaining value, it may be worth narrowing this to just that (with a test); otherwise it looks mostly superseded now. Not trying to discount the work here, just want to make sure we're not re-landing something.
   
   Thanks for the detailed audit @kaxil — really helpful context.
   
   You're right that #68779, #67935, and #68370 together cover the provider-layer symptom. Looking at what's left:
   
   **log.py reorder (Bug 1):** `getattr(remote, "processors")` was called before `dictConfig` ran, which meant `dictConfig`'s `_clearExistingHandlers` closed the just-built watchtower handler before any task log was emitted. With #68779's self-healing property this is no longer a live drop, the handler rebuilds on next access, but the reorder is still the correct root-cause fix and removes the need for the rebuild in the first place. It's a cleaner argument even if it's no longer strictly necessary for correctness.
   
   **supervisor.py `_close_remote_log_handler`:** You're right this now overlaps with `upload() → close()` from #67935/#68779. I'm happy to drop the supervisor.py changes entirely if you think log.py alone is worth keeping, or close the whole PR if you think it's fully superseded. I'd rather do the right thing for the codebase than land something redundant.
   
   What would you prefer?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3629015867


##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -133,6 +139,15 @@ def configure_logging(
         callsite_parameters=callsite_params,
     )
 
+    # dictConfig has now run — safe to build the watchtower handler.
+    # We append remote processors into the already-configured structlog chain,
+    # inserting before the final renderer (last processor in the chain).

Review Comment:
   Done. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3339279734


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -39,8 +41,6 @@
 from airflow.utils.log.logging_mixin import LoggingMixin
 
 if TYPE_CHECKING:
-    import structlog.typing

Review Comment:
   This should stay as a type checking only import?



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -467,7 +468,8 @@ def exit(n: int) -> NoReturn:
         except Exception as e:
             with suppress(Exception):
                 print(
-                    f"--- Last chance exception handler failed --- {repr(str(e))}\n", file=last_chance_stderr
+                    f"--- Last chance exception handler failed --- {repr(str(e))}\n",
+                    file=last_chance_stderr,

Review Comment:
   Formatting only change. Undo it



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -628,13 +630,22 @@ def start(
                     # execv replaces the process -- unreachable on success
                 else:
                     # Run the child entrypoint
-                    _fork_main(child_requests, child_stdout, child_stderr, child_logs.fileno(), target)
+                    _fork_main(
+                        child_requests,
+                        child_stdout,
+                        child_stderr,
+                        child_logs.fileno(),
+                        target,

Review Comment:
   Ditto



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2301,18 +2400,24 @@ def supervise_task(
 
         reset_secrets_masker()
 
+        log_ctx = _configure_logging(log_path, client) if log_path else contextlib.nullcontext((None, None))
+
         try:
-            process = ActivitySubprocess.start(
-                dag_rel_path=dag_rel_path,
-                what=ti,
-                client=client,
-                logger=logger,
-                bundle_info=bundle_info,
-                subprocess_logs_to_stdout=subprocess_logs_to_stdout,
-                sentry_integration=sentry_integration,
-            )
+            with log_ctx as (logger, _):

Review Comment:
   Why do we return two values if we only need one?



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2189,11 +2255,30 @@ def ensure_secrets_backend_loaded() -> list[BaseSecretsBackend]:
     return ensure_secrets_loaded(default_backends=fallback_backends)
 
 
-def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLogger, BinaryIO | TextIO]:
+def _close_remote_log_handler(handler: RemoteLogIO) -> None:
+    """
+    Close the remote log handler explicitly after all task log messages have been drained from the subprocess pipe.
+
+    Called after process.wait() returns, before process exit triggers
+    logging.shutdown(). This ensures Watchtower's internal batch queue
+    is flushed to CloudWatch before the process tears down.
+    """
+    with contextlib.suppress(Exception):

Review Comment:
   Hiding the error entirely makes it hard to debug problems. We should log it?



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2301,18 +2400,24 @@ def supervise_task(
 
         reset_secrets_masker()
 
+        log_ctx = _configure_logging(log_path, client) if log_path else contextlib.nullcontext((None, None))

Review Comment:
   Why did you move this? It changes what appears in the task logs by doing this later



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5774445350

   Thanks for the thorough review. Addressed the requested follow-up in c73bda4e0a:
   
   - Added the missing `getattr(remote, "processors", None)` default in `logging_processors()`.
   - Replaced the unrelated upload coverage with focused parametrized tests for processor ordering/placement and both `sending_to_supervisor` paths, plus coverage for handlers without `processors`.
   - Added `airflow-core/newsfragments/66633.bugfix.rst`.
   
   ---
   Drafted-by: Codex (GPT-5) (no human review before posting)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5972989967

   This one looks ready - with plenty of approvals and just request-changes @ashb -where all your asks seem to be handled. Shall we merge it ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3629018944


##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -133,6 +139,15 @@ def configure_logging(
         callsite_parameters=callsite_params,
     )
 
+    # dictConfig has now run — safe to build the watchtower handler.
+    # We append remote processors into the already-configured structlog chain,
+    # inserting before the final renderer (last processor in the chain).
+    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
+        current_processors = list(structlog.get_config()["processors"])
+        # Insert before the final renderer
+        updated_processors = current_processors[:-1] + list(remote_processors) + [current_processors[-1]]
+        structlog.configure(processors=updated_processors)

Review Comment:
   Done. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4950798335

   Addressed all outstanding review comments:
   
   - Reverted unrelated changes in `_handle_request` (GetXComCount ordering and send_msg parameter style)
   - Restored the dropped TODO comment in `_configure_logging`
   - Used walrus operator for `load_remote_log_handler()` check
   - Added `hasattr(handler, "close")` guard in `_close_remote_log_handler` to avoid mypy error after reverting the RemoteLogIO protocol change
   - Reverted all provider and shared/logging changes per @jason810496 guidance, this PR now touches only supervisor.py and log.py


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3237163268


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,29 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        # Defensive self-healing: if the handler was killed by logging.shutdown()
+        # (shutting_down=True), recreate it. This can happen if dictConfig() is called
+        # after the handler was first created, since dictConfig calls
+        # _clearExistingHandlers() -> logging.shutdown() on all existing handlers.
+        if self._handler is None or self._handler.shutting_down:
+            self._handler = self._create_handler()
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly init the handler to avoid infinite loops from logging during handler creation.
+        # We do NOT capture it in a closure variable — instead we access self.handler each time
+        # so that if the handler is killed by logging.shutdown() and recreated, the processor
+        # always uses the live instance rather than a dead one.#
+        _ = self.handler

Review Comment:
   The assignment itself doesn't matter, i think what matters is the access. Calling `self.handler` eagerly triggers the property which initializes the boto3 client and creates the watchtower handler before the processor closure is returned. Without this eager initialization, the first log emission would trigger handler creation, which itself logs internally, which triggers handler creation again, infinite loop.
   I've changed `_ = self.handler` to a bare `self.handler` with a comment making this clearer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb merged PR #66633:
URL: https://github.com/apache/airflow/pull/66633


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3629026471


##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -133,6 +139,15 @@ def configure_logging(
         callsite_parameters=callsite_params,
     )
 
+    # dictConfig has now run — safe to build the watchtower handler.
+    # We append remote processors into the already-configured structlog chain,
+    # inserting before the final renderer (last processor in the chain).
+    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
+        current_processors = list(structlog.get_config()["processors"])
+        # Insert before the final renderer
+        updated_processors = current_processors[:-1] + list(remote_processors) + [current_processors[-1]]
+        structlog.configure(processors=updated_processors)

Review Comment:
   Done. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5618831280

   @ashb can your review this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3237136547


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,29 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        # Defensive self-healing: if the handler was killed by logging.shutdown()
+        # (shutting_down=True), recreate it. This can happen if dictConfig() is called
+        # after the handler was first created, since dictConfig calls
+        # _clearExistingHandlers() -> logging.shutdown() on all existing handlers.

Review Comment:
   Bug 1 fixes the primary trigger, but the self-healing property in Bug 2 is defense-in-depth, any future dictConfig() call (from a plugin, test reconfiguration, or other code path) would re-kill a cached handler. Happy to remove it if the preference is to keep the fix minimal, but I'd argue it makes the system more resilient long-term. Open to guidance here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4658960146

   @korex-f — A reviewer (@ashb) has requested changes on this PR, so I've removed the `ready for maintainer review` label — the next step is on your side. Could you address the review comments (push a fix, or reply in-thread explaining why the feedback doesn't apply)? Once addressed, re-request review from @ashb or re-mark the PR ready and it returns to the maintainer queue. Thank you.
   
   _Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5004211732

   Done!
   All provider and shared/logging changes were already reverted in the latest push per @jason810496's guidance. This PR now touches only task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. The provider-side fix is being handled separately in #68779.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3443507349


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2405,9 +2422,30 @@ def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLog
 
     with _remote_logging_conn(client):
         processors = logging_processors(json_output=json_logs)
+
     logger = structlog.wrap_logger(underlying_logger, processors=processors, logger_name="task").bind()
 
-    return logger, log_file_descriptor
+    try:
+        yield logger
+    finally:
+        # Flush and close the remote handler now — AFTER the supervisor has
+        # drained all task log messages from the subprocess pipe (i.e. after
+        # process.wait() has returned).
+        #
+        # Without this, the only thing that ever closes the handler is
+        # Python's logging.shutdown() at process exit, which fires after
+        # supervise_task() returns. Any messages still queued in the handler
+        # at that point are silently dropped. For example, the AWS CloudWatch
+        # logger will emit:
+
+        # WatchtowerWarning: "Received message after logging system shutdown"
+        remote_handler = load_remote_log_handler()

Review Comment:
   Please keep the TODO instead of dropping it silently.
   ```suggestion
           # TODO: Use logging providers to handle the chunked upload for us etc.
           remote_handler = load_remote_log_handler()
   ```



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2385,11 +2382,31 @@ def ensure_secrets_backend_loaded() -> list[BaseSecretsBackend]:
     return ensure_secrets_loaded(default_backends=fallback_backends)
 
 
-def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLogger, BinaryIO | TextIO]:
+def _close_remote_log_handler(handler: RemoteLogIO) -> None:
+    """
+    Close the remote log handler explicitly after all task log messages have been drained from the subprocess pipe.
+
+    Called after process.wait() returns, before process exit triggers
+    logging.shutdown(). This ensures the remote handler's internal batch
+    queue is flushed before the process tears down. For example, the AWS
+    CloudWatch logger will emit:
+
+        WatchtowerWarning: "Received message after logging system shutdown"
+
+    if this is not done before process exit.
+    """
+    try:
+        handler.close()

Review Comment:
   It would be better to have more defensive compatible guard against the `handler.close` call.
   Since not all remote logging handler support `close` method when using the Task-SDK version including this patch.
   
   ```suggestion
           if hasattr(handler, "close", None):
               handler.close()
   ```



##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -30,6 +30,8 @@
 from typing import TYPE_CHECKING, Any
 
 import attrs
+import structlog
+import structlog.typing

Review Comment:
   ```suggestion
   ```



##########
shared/logging/src/airflow_shared/logging/remote.py:
##########
@@ -64,6 +64,10 @@ def read(self, relative_path: str, ti: RuntimeTI) -> LogResponse:
         """Read logs from the given remote log path."""
         ...
 
+    def close(self) -> None:
+        """Flush and close the remote log handler."""
+        ...

Review Comment:
   I don't think we need to introduce the `close` method at `RemoteLogIO` protocol in this PR.
   Since it's still nop for ES and OS remote IO implementation.



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2405,9 +2422,30 @@ def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLog
 
     with _remote_logging_conn(client):
         processors = logging_processors(json_output=json_logs)
+
     logger = structlog.wrap_logger(underlying_logger, processors=processors, logger_name="task").bind()
 
-    return logger, log_file_descriptor
+    try:
+        yield logger
+    finally:
+        # Flush and close the remote handler now — AFTER the supervisor has
+        # drained all task log messages from the subprocess pipe (i.e. after
+        # process.wait() has returned).
+        #
+        # Without this, the only thing that ever closes the handler is
+        # Python's logging.shutdown() at process exit, which fires after
+        # supervise_task() returns. Any messages still queued in the handler
+        # at that point are silently dropped. For example, the AWS CloudWatch
+        # logger will emit:
+
+        # WatchtowerWarning: "Received message after logging system shutdown"
+        remote_handler = load_remote_log_handler()
+        if remote_handler is not None:

Review Comment:
   nit:
   ```suggestion
           if (remote_handler := load_remote_log_handler()) is not None:
   ```



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -1774,8 +1773,6 @@ def _handle_request(self, msg: ToSupervisor, log: FilteringBoundLogger, req_id:
             dump_opts = {"exclude_unset": True}
         elif isinstance(msg, GetPrevSuccessfulDagRun):
             resp, dump_opts = handle_get_prev_successful_dag_run(self.client, self.id)
-        elif isinstance(msg, GetXComCount):
-            resp, dump_opts = handle_get_xcom_count(self.client, msg)

Review Comment:
   Could we revert these non-related changes? (the parameters style of `def send_msg(` and the order of `elif isinstance(msg, GetXComCount):` ).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3565598734


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2405,9 +2422,30 @@ def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLog
 
     with _remote_logging_conn(client):
         processors = logging_processors(json_output=json_logs)
+
     logger = structlog.wrap_logger(underlying_logger, processors=processors, logger_name="task").bind()
 
-    return logger, log_file_descriptor
+    try:
+        yield logger
+    finally:
+        # Flush and close the remote handler now — AFTER the supervisor has
+        # drained all task log messages from the subprocess pipe (i.e. after
+        # process.wait() has returned).
+        #
+        # Without this, the only thing that ever closes the handler is
+        # Python's logging.shutdown() at process exit, which fires after
+        # supervise_task() returns. Any messages still queued in the handler
+        # at that point are silently dropped. For example, the AWS CloudWatch
+        # logger will emit:
+
+        # WatchtowerWarning: "Received message after logging system shutdown"
+        remote_handler = load_remote_log_handler()

Review Comment:
   Applied suggestion. Thanks
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3237148125


##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -118,8 +118,14 @@ def configure_logging(
     if mask_secrets:
         extra_processors += (mask_logs,)
 
-    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
-        extra_processors += remote_processors
+    # NOTE: Do NOT call getattr(remote, "processors") here.
+    # Accessing remote.processors triggers creation of the watchtower CloudWatchLogHandler

Review Comment:
   Thanks. I have made it generic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5032508137

   > > > Can the provider related changes be split into a dedicated PR?
   > > 
   > > 
   > > Done! All provider and shared/logging changes were already reverted in the latest push per @jason810496's guidance. This PR now touches only task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. The provider-side fix is being handled separately in #68779.
   > 
   > You may not have pushed that, there are still provider changes in the current diff:
   > 
   > ```
   >   diff --git a/providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
   >   -    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
   >   +    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
   >   
   >   diff --git a/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
   >   -    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
   >   +    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
   >   +        if ti is None:
   >   +            return
   >   
   >   diff --git a/providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py
   >   -    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
   >   +    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
   >   +        if ti is None:
   >   +            return
   > ```
   > 
   > Edit to add: the whitespace change in supervisor.py that you said was done is also not there.
   
   Sorry for the confusion, the revert was there but spread across multiple commits, making it look like provider changes were still present in the per-commit view. The net diff against upstream/main now only touches task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. No provider files are modified.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5038085928

   Picking up on @o-nikolas's merge question -- I think this needs a second look at scope before it goes in, because a fair bit of it landed via other PRs while this was open:
   
   - **#68779** (merged) made the CloudWatch `RemoteLogIO` self-healing: the `handler` property rebuilds if `dictConfig` closed it mid-task, and `upload()` now calls `close()` to flush pending events. That covers the #66475 symptom at the provider layer.
   - **#67935** (merged) moved `_upload_logs()` into a `finally` after `update_task_state_if_needed()`, so logs upload even when the state update raises.
   - **#68370** (merged) fixed the cached-`None`-connection case so the worker can upload at all.
   
   Given that, the supervisor.py side here (explicit `_close_remote_log_handler` on teardown) now overlaps with `_upload_logs()` → `upload()` → `close()`, which already runs on the state-update error path after #67935. The log.py reorder (build the handler after `dictConfig`) is the one piece not on main, but with the self-healing handler it's no longer a live drop -- more of a cleaner-root-cause argument.
   
   @korex-f -- is there a failure mode left that the three merged PRs don't cover? If the log.py reorder is the remaining value, it may be worth narrowing this to just that (with a test); otherwise it looks mostly superseded now. Not trying to discount the work here, just want to make sure we're not re-landing something.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3230014314


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,29 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        # Defensive self-healing: if the handler was killed by logging.shutdown()
+        # (shutting_down=True), recreate it. This can happen if dictConfig() is called
+        # after the handler was first created, since dictConfig calls
+        # _clearExistingHandlers() -> logging.shutdown() on all existing handlers.
+        if self._handler is None or self._handler.shutting_down:
+            self._handler = self._create_handler()
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly init the handler to avoid infinite loops from logging during handler creation.
+        # We do NOT capture it in a closure variable — instead we access self.handler each time
+        # so that if the handler is killed by logging.shutdown() and recreated, the processor
+        # always uses the live instance rather than a dead one.#

Review Comment:
   ```suggestion
           # always uses the live instance rather than a dead one.
   ```



##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,29 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        # Defensive self-healing: if the handler was killed by logging.shutdown()
+        # (shutting_down=True), recreate it. This can happen if dictConfig() is called
+        # after the handler was first created, since dictConfig calls
+        # _clearExistingHandlers() -> logging.shutdown() on all existing handlers.

Review Comment:
   Do we still need to be this defensive if the other issues are fixed?



##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,29 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        # Defensive self-healing: if the handler was killed by logging.shutdown()
+        # (shutting_down=True), recreate it. This can happen if dictConfig() is called
+        # after the handler was first created, since dictConfig calls
+        # _clearExistingHandlers() -> logging.shutdown() on all existing handlers.
+        if self._handler is None or self._handler.shutting_down:
+            self._handler = self._create_handler()
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly init the handler to avoid infinite loops from logging during handler creation.
+        # We do NOT capture it in a closure variable — instead we access self.handler each time
+        # so that if the handler is killed by logging.shutdown() and recreated, the processor
+        # always uses the live instance rather than a dead one.#
+        _ = self.handler

Review Comment:
   Why bother assign this to anything at all? Is that needed?



##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -118,8 +118,14 @@ def configure_logging(
     if mask_secrets:
         extra_processors += (mask_logs,)
 
-    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
-        extra_processors += remote_processors
+    # NOTE: Do NOT call getattr(remote, "processors") here.
+    # Accessing remote.processors triggers creation of the watchtower CloudWatchLogHandler

Review Comment:
   Again make this generic, or make it clear you're using the AWS Cloudwatch handler as an example of what can happen. There are many other provider handlers not just Cloudwatch.  



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########


Review Comment:
   @ashb or @amoghrajesh either one of you should have a look at these changes.



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2209,9 +2235,29 @@ def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLog
 
     with _remote_logging_conn(client):
         processors = logging_processors(json_output=json_logs)
+
     logger = structlog.wrap_logger(underlying_logger, processors=processors, logger_name="task").bind()
 
-    return logger, log_file_descriptor
+    try:
+        yield logger, log_file_descriptor
+    finally:
+        # Flush and close the remote handler now — AFTER the supervisor has
+        # drained all task log messages from the subprocess pipe (i.e. after
+        # process.wait() has returned).
+        #
+        # Without this, the only thing that ever closes the handler is
+        # Python's logging.shutdown() at process exit, which fires after
+        # supervise_task() returns. Any messages still queued in the handler
+        # at that point are silently dropped, producing:
+        #
+        # WatchtowerWarning: "Received message after logging system shutdown"

Review Comment:
   We shouldn't just mention exceptions and warnings that providers throw in core/sdk code without providing the context that this is just one example of what one provider based logger will show.
   
   ```suggestion
           # at that point are silently dropped. The AWS Cloudwatch logger for example will emit:
           #
           # WatchtowerWarning: "Received message after logging system shutdown"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3565563440


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2385,11 +2382,31 @@ def ensure_secrets_backend_loaded() -> list[BaseSecretsBackend]:
     return ensure_secrets_loaded(default_backends=fallback_backends)
 
 
-def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLogger, BinaryIO | TextIO]:
+def _close_remote_log_handler(handler: RemoteLogIO) -> None:
+    """
+    Close the remote log handler explicitly after all task log messages have been drained from the subprocess pipe.
+
+    Called after process.wait() returns, before process exit triggers
+    logging.shutdown(). This ensures the remote handler's internal batch
+    queue is flushed before the process tears down. For example, the AWS
+    CloudWatch logger will emit:
+
+        WatchtowerWarning: "Received message after logging system shutdown"
+
+    if this is not done before process exit.
+    """
+    try:
+        handler.close()

Review Comment:
   Applied suggestion. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3699844216


##########
task-sdk/tests/task_sdk/test_log.py:
##########
@@ -0,0 +1,166 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import structlog
+import structlog.testing
+from uuid6 import uuid7
+
+from airflow.sdk import log as sdk_log
+
+
+def _make_ti():
+    ti = mock.MagicMock()
+    ti.id = uuid7()
+    return ti
+
+
+def _make_logger():
+    """Build a FilteringBoundLogger-like object exposing ``_logger``."""
+    logger = mock.MagicMock()
+    logger._logger = mock.MagicMock()
+    return logger
+
+
+class TestUploadToRemote:
+    def test_warns_when_handler_unavailable(self):
+        ti = _make_ti()
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=None),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_handler_unavailable"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+
+    def test_warns_when_path_resolution_fails(self):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        boom = RuntimeError("cannot resolve path")
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", side_effect=boom),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_path_resolution_failed"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+        assert events[0]["exc_info"] is boom
+        handler.upload.assert_not_called()
+
+    def test_warns_when_upload_fails(self, tmp_path):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        boom = RuntimeError("s3 unreachable")
+        handler.upload.side_effect = boom
+        relative = tmp_path / "dag_id" / "run_id" / "task.log"
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", return_value=relative),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_upload_failed"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+        assert events[0]["log_relative_path"] == relative.as_posix()
+        assert events[0]["exc_info"] is boom

Review Comment:
   Done. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3363901215


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2301,18 +2400,24 @@ def supervise_task(
 
         reset_secrets_masker()
 
+        log_ctx = _configure_logging(log_path, client) if log_path else contextlib.nullcontext((None, None))
+
         try:
-            process = ActivitySubprocess.start(
-                dag_rel_path=dag_rel_path,
-                what=ti,
-                client=client,
-                logger=logger,
-                bundle_info=bundle_info,
-                subprocess_logs_to_stdout=subprocess_logs_to_stdout,
-                sentry_integration=sentry_integration,
-            )
+            with log_ctx as (logger, _):

Review Comment:
   My bad, fixed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4766682314

   Thanks for sharing #68779 @jason810496, having read through it, I agree it's the cleaner approach to the provider side. The _closed flag correctly distinguishes dictConfig mid-task closures from genuine teardown, which is something my approach didn't handle.
   
   Given this, I'd like to scope this PR down to task-sdk only:
   - Issue 1 (log.py): getattr(remote, "processors") called before dictConfig runs — #68779 doesn't fix this and it's the root cause 
     that makes Bug 2 worse
   - Issue 3 (supervisor.py): _close_remote_log_handler context manager to ensure the handler is flushed after process.wait() returns
   
   That means reverting all provider changes from this PR and letting #68779 carry those.
   
   Does that scope make sense @ashb? Happy to also drop the close() protocol addition from RemoteLogIO per your comment, since #68779 handles the provider side without needing it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5089383379

   > Thank you for the update.
   > 
   > > is there a failure mode left that the three merged PRs don't cover?
   > 
   > Yes `log.py reorder (Bug 1): getattr(remote, "processors") was called before dictConfig ran, which meant dictConfig's _clearExistingHandlers closed the just-built watchtower handler before any task log was emitted.` statement is correct.
   > 
   > I'd like to keep the only the `log.py` changes as that is the key point to resolve the unexpected `.close` on logging handler. The `supervisors.py` change is more "nice to have" but it _might_ introduce further regression. Let's keep the change minimal if possible, thanks.
   
   Thanks for the clarity. Reverting all supervisor.py changes now,  this PR will only contain the log.py reorder (Bug 1). supervisor.py can be revisited in a follow-up if needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3565614774


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -1774,8 +1773,6 @@ def _handle_request(self, msg: ToSupervisor, log: FilteringBoundLogger, req_id:
             dump_opts = {"exclude_unset": True}
         elif isinstance(msg, GetPrevSuccessfulDagRun):
             resp, dump_opts = handle_get_prev_successful_dag_run(self.client, self.id)
-        elif isinstance(msg, GetXComCount):
-            resp, dump_opts = handle_get_xcom_count(self.client, msg)

Review Comment:
   Reverted the changes. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4476939509

   @korex-f — There are 2 unresolved review threads on this PR from @o-nikolas. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? Once you believe the feedback is addressed, mark the thread as resolved so the reviewer isn't re-pinged needlessly. Thanks!
   
   ---
   
   _Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3328318080


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,38 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    _handler_creating: bool = attrs.field(default=False, init=False, repr=False)
+
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        if self._handler_creating:
+            # Re-entrant call during handler creation, some libraries log internally
+            # during initialization, which triggers this process again before
+            # handler is fully constructed.
+            if self._handler is not None and isinstance(self._handler, watchtower.CloudWatchLogHandler):
+                return self._handler
+            raise structlog.DropEvent()
+        if self._handler is None or self._handler.shutting_down:
+            self._handler_creating = True
+            try:
+                self._handler = self._create_handler()
+            finally:
+                self._handler_creating = False
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly initialize the handler before the closure is created.
+        # Without this, the first log emission would trigger handler creation,
+        # which itself logs, causing an infinite loop.
+        self._handler

Review Comment:
   > I could be mistaken, but shouldn't this be hitting the new `self.handler` property instead of the underlying `self._handler` field to make sure it gets init'd?
   
   You're absolutely right, that's a bug. self._handler directly accesses the backing field which is None at that point, so the eager init is currently a no-op and the re-entrancy guard never fires during initialization. I would change this to self.handler now. Thanks for catching this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3306775272


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,38 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    _handler_creating: bool = attrs.field(default=False, init=False, repr=False)
+
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        if self._handler_creating:
+            # Re-entrant call during handler creation, some libraries log internally
+            # during initialization, which triggers this process again before
+            # handler is fully constructed.
+            if self._handler is not None and isinstance(self._handler, watchtower.CloudWatchLogHandler):
+                return self._handler
+            raise structlog.DropEvent()
+        if self._handler is None or self._handler.shutting_down:
+            self._handler_creating = True
+            try:
+                self._handler = self._create_handler()
+            finally:
+                self._handler_creating = False
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly initialize the handler before the closure is created.
+        # Without this, the first log emission would trigger handler creation,
+        # which itself logs, causing an infinite loop.
+        self._handler

Review Comment:
   I could be mistaken, but shouldn't this be hitting the new `self.handler` property instead of the underlying `self._handler` field to make sure it gets init'd?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3227419819


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2189,11 +2190,36 @@ def ensure_secrets_backend_loaded() -> list[BaseSecretsBackend]:
     return ensure_secrets_loaded(default_backends=fallback_backends)
 
 
-def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLogger, BinaryIO | TextIO]:
+def _close_remote_log_handler(handler: RemoteLogIO) -> None:
+    """
+    Close the remote log handler explicitly after all task log messages have been drained, before process exit triggers logging.shutdown().
+
+    This prevents WatchtowerWarning: "Received message after logging system shutdown" which causes CloudWatch log streams to be silently truncated.
+    """
+    import logging as _logging
+
+    # If the handler is also a stdlib logging.Handler, deregister it from
+    # the logging hierarchy so logging.shutdown() at process exit cannot
+    # double-close it.
+    if isinstance(handler, _logging.Handler):
+        root = _logging.getLogger()

Review Comment:
   Is this `isinstance(handler, _logging.Handler)` guard expected to ever evaluate truthy in practice? Looking at the three concrete `RemoteLogIO` implementations:
   
   - `CloudWatchRemoteLogIO(LoggingMixin)` in `providers/amazon/.../cloudwatch_task_handler.py:85` — `LoggingMixin` just provides a `.log` property, not a `logging.Handler` base
   - `ElasticsearchRemoteLogIO` and `OpenSearchRemoteLogIO` are the same shape (`LoggingMixin`, not `Handler`)
   
   So the dereg block (`root.removeHandler` / per-logger `removeHandler`) is effectively dead, and the only thing this function does at runtime is `handler.close()` — which for CloudWatch lands on the existing `close()` that just calls `self.handler.flush()` without touching the inner watchtower handler's stdlib registration. If the goal is to prevent `logging.shutdown()` from later double-closing the *inner* watchtower handler, you probably want one level deeper — something like `getattr(handler, 'handler', None)` or letting each `RemoteLogIO.close()` impl handle its own dereg (and document that contract on the protocol).
   
   Happy to be wrong if there's a planned `RemoteLogIO` subclass that actually inherits `logging.Handler` — wasn't sure from the diff.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4421261424

   You're very much welcome.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5007983638

   @jason810496 and @ferruzzi, does this look good to approve/merge for y'all?
   
   @ashb it looks like your feedback was addressed, but you "requested changes" which makes me think you saw something very serious, are you happy with this PR now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3245073304


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,29 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        # Defensive self-healing: if the handler was killed by logging.shutdown()
+        # (shutting_down=True), recreate it. This can happen if dictConfig() is called
+        # after the handler was first created, since dictConfig calls
+        # _clearExistingHandlers() -> logging.shutdown() on all existing handlers.
+        if self._handler is None or self._handler.shutting_down:
+            self._handler = self._create_handler()
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly init the handler to avoid infinite loops from logging during handler creation.
+        # We do NOT capture it in a closure variable — instead we access self.handler each time
+        # so that if the handler is killed by logging.shutdown() and recreated, the processor
+        # always uses the live instance rather than a dead one.#
+        _ = self.handler

Review Comment:
   Yupp, wasn't questioning the access, just the assignment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5172383167

   @kaxil 
   Thanks for the detailed review, i have addressed all three points:
   - Dropped the three test_warns_* tests and reverted upload_to_remote 
     to the simple version
   - Removed the cached_property mention from the log.py comment
   - Fixed TestConfigureLogging to save and restore global structlog 
     processor state in a finally block and clear the cache on exit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5149814844

   > Hi @korex-f,
   > 
   > Thanks for follow-up on this. We need to do the following items before merge.
   > 
   >     * Add unit test to exercise the patch itself
   > 
   >     * Fix the CI failure
   > 
   >     * Rephrase the PR title and the description
   > 
   > 
   > Thanks.
   Added a regression test in test_log.py that verifies load_remote_log_handler() is called only after the inner configure_logging() (dictConfig) returns. The CI failure in test_inject_parent_job_info_with_resume_on_retry is unrelated, it's a Glue operator test making a real network call that times out, last touched in #64513. Happy to re-trigger CI to confirm it's a flake.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3698273347


##########
task-sdk/tests/task_sdk/test_log.py:
##########
@@ -0,0 +1,166 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import structlog
+import structlog.testing
+from uuid6 import uuid7
+
+from airflow.sdk import log as sdk_log
+
+
+def _make_ti():
+    ti = mock.MagicMock()
+    ti.id = uuid7()
+    return ti
+
+
+def _make_logger():
+    """Build a FilteringBoundLogger-like object exposing ``_logger``."""
+    logger = mock.MagicMock()
+    logger._logger = mock.MagicMock()
+    return logger
+
+
+class TestUploadToRemote:
+    def test_warns_when_handler_unavailable(self):
+        ti = _make_ti()
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=None),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_handler_unavailable"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+
+    def test_warns_when_path_resolution_fails(self):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        boom = RuntimeError("cannot resolve path")
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", side_effect=boom),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_path_resolution_failed"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+        assert events[0]["exc_info"] is boom

Review Comment:
   nit: How about asserting with `events[0] == {"log_level": ..., "ti_id": ...}` style?



##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -119,8 +119,14 @@ def configure_logging(
     if mask_secrets:
         extra_processors += (mask_logs,)
 
-    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
-        extra_processors += remote_processors
+    # NOTE: Do NOT call getattr(remote, "processors") here.
+    # Accessing remote.processors triggers creation of the remote handler
+    # via a cached_property. The configure_logging() call below runs dictConfig() internally,

Review Comment:
   We don't need to mentioned the `cached_property` workaround anymore.
   
   ```suggestion
       # The configure_logging() call below runs dictConfig() internally,
   ```



##########
task-sdk/tests/task_sdk/test_log.py:
##########
@@ -0,0 +1,166 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import structlog
+import structlog.testing
+from uuid6 import uuid7
+
+from airflow.sdk import log as sdk_log
+
+
+def _make_ti():
+    ti = mock.MagicMock()
+    ti.id = uuid7()
+    return ti
+
+
+def _make_logger():
+    """Build a FilteringBoundLogger-like object exposing ``_logger``."""
+    logger = mock.MagicMock()
+    logger._logger = mock.MagicMock()
+    return logger
+
+
+class TestUploadToRemote:
+    def test_warns_when_handler_unavailable(self):
+        ti = _make_ti()
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=None),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_handler_unavailable"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+
+    def test_warns_when_path_resolution_fails(self):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        boom = RuntimeError("cannot resolve path")
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", side_effect=boom),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_path_resolution_failed"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+        assert events[0]["exc_info"] is boom
+        handler.upload.assert_not_called()
+
+    def test_warns_when_upload_fails(self, tmp_path):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        boom = RuntimeError("s3 unreachable")
+        handler.upload.side_effect = boom
+        relative = tmp_path / "dag_id" / "run_id" / "task.log"
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", return_value=relative),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_upload_failed"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+        assert events[0]["log_relative_path"] == relative.as_posix()
+        assert events[0]["exc_info"] is boom

Review Comment:
   nit: Ditto.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r4064350912


##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -134,6 +139,22 @@ def configure_logging(
         callsite_parameters=callsite_params,
     )
 
+    # dictConfig has now run, so it is safe to build the remote handler. Re-inject the remote
+    # processors into the global structlog chain (before the final renderer) for parity with the old
+    # extra_processors layout. Task-log streaming itself does not rely on this: it uses the
+    # file-backed logger built from logging_processors(), which loads remote.processors lazily.
+    if (
+        not sending_to_supervisor
+        and (remote := load_remote_log_handler())
+        and (remote_processors := getattr(remote, "processors", None))

Review Comment:
   This is the half that got fixed. The identical line in `logging_processors()` didn't:
   
   ```python
   77:    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
   ```
   
   Your description says the PR adds *"a `None` default to `getattr` to handle third-party `RemoteLogIO` objects missing the `processors` attribute"* — but `logging_processors()` is reached from `supervisor.py:2593`, `callback_supervisor.py:391`, `dag_processing/manager.py:1437` and `triggerer_job_runner.py:1035`, so a third-party handler without `.processors` still raises `AttributeError` in the supervisor, the Dag processor and the triggerer. Those are arguably the paths where it matters most.
   
   One token (`, None`) at line 77 and the claim in the description becomes true.



##########
task-sdk/tests/task_sdk/test_log.py:
##########
@@ -0,0 +1,122 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import structlog
+import structlog.testing
+from uuid6 import uuid7
+
+from airflow.sdk import log as sdk_log
+
+
+def _make_ti():
+    ti = mock.MagicMock()
+    ti.id = uuid7()
+    return ti
+
+
+def _make_logger():
+    """Build a FilteringBoundLogger-like object exposing ``_logger``."""
+    logger = mock.MagicMock()
+    logger._logger = mock.MagicMock()
+    return logger
+
+
+class TestUploadToRemote:

Review Comment:
   `upload_to_remote` is at `log.py:232` and this PR doesn't touch it; it's also already covered indirectly by `test_supervisor.py` and `test_callback_supervisor.py`. Per `AGENTS.md` § Testing Standards: *"Do not add tests for pre-existing logic that was already present before the PR."*
   
   I realise the pull here is that you're creating `test_log.py` from scratch and it feels bare with one test in it — but scope is scope, and the effort is better spent on the two gaps in my comment on line 119.
   
   Style nits while you're in the file, both from the same section: `@mock.patch` decorators are preferred over `with mock.patch(...)` context managers, and mocks should carry `spec`/`autospec` — `mock.MagicMock()` for the handler, the TI and the logger will accept any attribute you ask of it, including ones the real object doesn't have.



##########
task-sdk/tests/task_sdk/test_log.py:
##########
@@ -0,0 +1,122 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import structlog
+import structlog.testing
+from uuid6 import uuid7
+
+from airflow.sdk import log as sdk_log
+
+
+def _make_ti():
+    ti = mock.MagicMock()
+    ti.id = uuid7()
+    return ti
+
+
+def _make_logger():
+    """Build a FilteringBoundLogger-like object exposing ``_logger``."""
+    logger = mock.MagicMock()
+    logger._logger = mock.MagicMock()
+    return logger
+
+
+class TestUploadToRemote:
+    def test_silent_when_relative_path_is_none(self):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", return_value=None),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        assert captured == []
+        handler.upload.assert_not_called()
+
+    def test_silent_on_success(self, tmp_path):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        relative = tmp_path / "dag_id" / "run_id" / "task.log"
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", return_value=relative),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        assert captured == []
+        handler.upload.assert_called_once_with(relative.as_posix(), ti)
+
+
+class TestConfigureLogging:
+    def test_remote_processors_injected_after_dictconfig(self):
+        """
+        Regression test: remote processor injection must happen AFTER dictConfig() runs.
+
+        dictConfig()'s non-incremental reset closes every handler in
+        logging._handlerList. If the remote handler is built before dictConfig
+        runs, it is closed before any task log is emitted and silently drops
+        all records.
+        """
+        import airflow.sdk._shared.logging as shared_logging
+
+        call_order = []
+
+        mock_handler = mock.MagicMock()
+        mock_handler.processors = (mock.MagicMock(),)
+
+        def track_load_remote():
+            call_order.append("load_remote_log_handler")
+            return mock_handler
+
+        original_inner = shared_logging.configure_logging
+
+        def track_inner_configure(*args, **kwargs):
+            call_order.append("dictConfig")
+            return original_inner(*args, **kwargs)
+
+        # Save global structlog state so we can restore it after the test
+        original_processors = list(structlog.get_config()["processors"])
+
+        sdk_log.configure_logging.cache_clear()
+
+        try:
+            with (
+                mock.patch.object(sdk_log, "load_remote_log_handler", side_effect=track_load_remote),
+                mock.patch.object(shared_logging, "configure_logging", side_effect=track_inner_configure),
+            ):
+                sdk_log.configure_logging()
+        finally:
+            # Restore global structlog processor chain and clear the cache so
+            # subsequent tests start from a clean state
+            structlog.configure(processors=original_processors)
+            sdk_log.configure_logging.cache_clear()
+
+        assert "dictConfig" in call_order, "inner configure_logging was never called"
+        assert "load_remote_log_handler" in call_order, "load_remote_log_handler() was never called"
+        dictconfig_pos = call_order.index("dictConfig")
+        load_remote_pos = call_order.index("load_remote_log_handler")
+        assert dictconfig_pos < load_remote_pos, (

Review Comment:
   This assertion is the weakest property the test could have checked. It confirms `load_remote_log_handler` was *called* after `dictConfig`, and nothing else.
   
   Two consequences:
   
   - The insertion at `log.py:155` — `current_processors[:-1] + list(remote_processors) + [current_processors[-1]]` — is the fiddliest line in the diff and is completely unpinned. This test passes just as happily if the processors are appended *after* the renderer, where they'd silently never run. Asserting the mock processor's presence and position in `structlog.get_config()["processors"]` would catch that, and it's barely more code than you already have.
   - The `not sending_to_supervisor` gate at `log.py:147` is new behaviour, not a move, and nothing exercises it. Inverting or dropping it would be caught by nothing. Per `AGENTS.md` § Testing Standards, *"Every changed or added behaviour must have a test"* — a parametrized case over `sending_to_supervisor=True/False` would cover it.



##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -119,8 +119,13 @@ def configure_logging(
     if mask_secrets:
         extra_processors += (mask_logs,)
 
-    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
-        extra_processors += remote_processors
+    # NOTE: Do NOT call getattr(remote, "processors") here.

Review Comment:
   Two things about this comment block.
   
   First, it explains code that is no longer here, and then the same explanation is given again at line 142. A regression guard is legitimate context — *"don't move this back above `configure_logging()`"* is worth recording — but one of the two blocks should carry it, not both. `AGENTS.md` § Coding Standards: *"Do not narrate code, repeat code."*
   
   Second, a question about the last line: *"Remote processors are injected AFTER dictConfig"* — true here, but `logging_processors()` at line 62 is the same bug's other door. It's `@cache`d and its first call runs `load_remote_log_handler()` → `_load_logging_config()`, which builds the handler and registers it in `logging._handlerList`. Anything reaching it before this function's `dictConfig()` gets the original bug back. I checked the four callers and they all run after startup configuration, so I don't think it's live — but nothing records that invariant, and your comment calling that path "lazy" reads as if it's inherently safe. Lazy only means "on first call".



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3258404833


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,29 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        # Defensive self-healing: if the handler was killed by logging.shutdown()
+        # (shutting_down=True), recreate it. This can happen if dictConfig() is called
+        # after the handler was first created, since dictConfig calls
+        # _clearExistingHandlers() -> logging.shutdown() on all existing handlers.
+        if self._handler is None or self._handler.shutting_down:
+            self._handler = self._create_handler()
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly init the handler to avoid infinite loops from logging during handler creation.
+        # We do NOT capture it in a closure variable — instead we access self.handler each time
+        # so that if the handler is killed by logging.shutdown() and recreated, the processor
+        # always uses the live instance rather than a dead one.#

Review Comment:
   Thanks for this catch! I remove the needless #



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3363897689


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2301,18 +2400,24 @@ def supervise_task(
 
         reset_secrets_masker()
 
+        log_ctx = _configure_logging(log_path, client) if log_path else contextlib.nullcontext((None, None))

Review Comment:
   Fixed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5026970518

   > > Can the provider related changes be split into a dedicated PR?
   > 
   > Done! All provider and shared/logging changes were already reverted in the latest push per @jason810496's guidance. This PR now touches only task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. The provider-side fix is being handled separately in #68779.
   
   You may not have pushed that, there are still provider changes in the current diff:
   
   ```
     diff --git a/providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
     -    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
     +    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
     
     diff --git a/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
     -    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
     +    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
     +        if ti is None:
     +            return
     
     diff --git a/providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py
     -    def upload(self, path: os.PathLike | str, ti: RuntimeTI):
     +    def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
     +        if ti is None:
     +            return
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5189646584

   > Nice!
   
   Thanks for the guidance. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3621177953


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -883,11 +856,7 @@ def _serialize_response(self, msg: BaseModel | ErrorResponse, **dump_opts) -> di
         return msg.model_dump(**dump_opts)
 
     def send_msg(
-        self,
-        msg: BaseModel | None,
-        request_id: int,
-        error: ErrorResponse | None = None,
-        **dump_opts,
+        self, msg: BaseModel | None, request_id: int, error: ErrorResponse | None = None, **dump_opts
     ):

Review Comment:
   Would prefer not change this part.



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2437,14 +2423,36 @@ def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLog
 
     with _remote_logging_conn(client):
         processors = logging_processors(json_output=json_logs)
+
     logger = structlog.wrap_logger(underlying_logger, processors=processors, logger_name="task").bind()
 
-    return logger, log_file_descriptor
+    try:
+        yield logger
+    finally:
+        # Flush and close the remote handler now — AFTER the supervisor has
+        # drained all task log messages from the subprocess pipe (i.e. after
+        # process.wait() has returned).
+        #
+        # Without this, the only thing that ever closes the handler is
+        # Python's logging.shutdown() at process exit, which fires after
+        # supervise_task() returns. Any messages still queued in the handler
+        # at that point are silently dropped. For example, the AWS CloudWatch
+        # logger will emit:
+
+        # WatchtowerWarning: "Received message after logging system shutdown"
+
+        # TODO: Use logging providers to handle the chunked upload for us etc.
+        if (remote_handler := load_remote_log_handler()) is not None:
+            _close_remote_log_handler(remote_handler)
+
+        if log_file_descriptor is not None:
+            with contextlib.suppress(Exception):
+                log_file_descriptor.close()

Review Comment:
   Should we switch the order of closing `log_file_descriptor` and the `remote log handler`? IIUC, closing the file first then flush the renaming logs to remote IO makes more sense to me.



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2437,14 +2423,36 @@ def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLog
 
     with _remote_logging_conn(client):
         processors = logging_processors(json_output=json_logs)
+
     logger = structlog.wrap_logger(underlying_logger, processors=processors, logger_name="task").bind()
 
-    return logger, log_file_descriptor
+    try:
+        yield logger
+    finally:
+        # Flush and close the remote handler now — AFTER the supervisor has
+        # drained all task log messages from the subprocess pipe (i.e. after
+        # process.wait() has returned).
+        #
+        # Without this, the only thing that ever closes the handler is
+        # Python's logging.shutdown() at process exit, which fires after
+        # supervise_task() returns. Any messages still queued in the handler
+        # at that point are silently dropped. For example, the AWS CloudWatch
+        # logger will emit:
+
+        # WatchtowerWarning: "Received message after logging system shutdown"
+
+        # TODO: Use logging providers to handle the chunked upload for us etc.
+        if (remote_handler := load_remote_log_handler()) is not None:
+            _close_remote_log_handler(remote_handler)

Review Comment:
   The scope and the purpose of the function is small and clear enough, we don't need the additional function for this. I'd like to collapse it back.
   ```suggestion
               try:
                   if hasattr(handler, "close"):
                       handler.close()
               except Exception:
                   log.warning("Failed to close remote log handler", exc_info=True)
   ```



##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -133,6 +139,15 @@ def configure_logging(
         callsite_parameters=callsite_params,
     )
 
+    # dictConfig has now run — safe to build the watchtower handler.
+    # We append remote processors into the already-configured structlog chain,
+    # inserting before the final renderer (last processor in the chain).
+    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
+        current_processors = list(structlog.get_config()["processors"])
+        # Insert before the final renderer
+        updated_processors = current_processors[:-1] + list(remote_processors) + [current_processors[-1]]
+        structlog.configure(processors=updated_processors)

Review Comment:
   Unlike the old `extra_processors` path, stdlib-routed records (`ProcessorFormatter`'s `foreign_pre_chain`) no longer pass through the remote processors. Neutral today — `ProcessorFormatter` invokes processors with `logger=None`, so the CloudWatch proc skipped them anyway — but worth documenting so the asymmetry isn't "fixed" later:
   
   ```suggestion
           # NOTE: unlike the old extra_processors path, stdlib-routed records (ProcessorFormatter's
           # foreign_pre_chain) intentionally do NOT pass through the remote processors.
           structlog.configure(processors=updated_processors)
   ```
   



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2417,11 +2382,32 @@ def ensure_secrets_backend_loaded() -> list[BaseSecretsBackend]:
     return ensure_secrets_loaded(default_backends=fallback_backends)
 
 
-def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLogger, BinaryIO | TextIO]:
+def _close_remote_log_handler(handler: RemoteLogIO) -> None:
+    """
+    Close the remote log handler explicitly after all task log messages have been drained from the subprocess pipe.
+
+    Called after process.wait() returns, before process exit triggers
+    logging.shutdown(). This ensures the remote handler's internal batch
+    queue is flushed before the process tears down. For example, the AWS
+    CloudWatch logger will emit:
+
+        WatchtowerWarning: "Received message after logging system shutdown"
+
+    if this is not done before process exit.
+    """
+    try:
+        if hasattr(handler, "close"):
+            handler.close()
+    except Exception:
+        log.warning("Failed to close remote log handler", exc_info=True)

Review Comment:
   ```suggestion
   ```



##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -133,6 +139,15 @@ def configure_logging(
         callsite_parameters=callsite_params,
     )
 
+    # dictConfig has now run — safe to build the watchtower handler.
+    # We append remote processors into the already-configured structlog chain,
+    # inserting before the final renderer (last processor in the chain).
+    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
+        current_processors = list(structlog.get_config()["processors"])
+        # Insert before the final renderer
+        updated_processors = current_processors[:-1] + list(remote_processors) + [current_processors[-1]]
+        structlog.configure(processors=updated_processors)

Review Comment:
   Two hardening points, both cheap while this is being rewritten:
   
   1. This also runs in the task subprocess (`_configure_logs_over_json_channel` → `configure_logging(..., sending_to_supervisor=True)`), whose chain deliberately **omits `mask_logs`** — masking happens supervisor-side. Any future `RemoteLogIO` whose processors don't require a file-backed logger would ship **unmasked** events from the subprocess *and* duplicate them (the supervisor re-logs the same events through its own remote-processor chain). It's inert for CloudWatch today, but this is exactly the trap the old code had, so let's gate it while we're here.
   2. `getattr(remote, "processors")` without a default raises `AttributeError` for a third-party `REMOTE_TASK_LOG` object missing the property, and that takes down logging setup for the whole process (same pre-existing pattern in `logging_processors`).
   
   ```suggestion
       if (
           not sending_to_supervisor
           and (remote := load_remote_log_handler())
           and (remote_processors := getattr(remote, "processors", None))
       ):
           current_processors = list(structlog.get_config()["processors"])
           # Insert before the final renderer
           updated_processors = current_processors[:-1] + list(remote_processors) + [current_processors[-1]]
           structlog.configure(processors=updated_processors)
   ```
   



##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -133,6 +139,15 @@ def configure_logging(
         callsite_parameters=callsite_params,
     )
 
+    # dictConfig has now run — safe to build the watchtower handler.
+    # We append remote processors into the already-configured structlog chain,
+    # inserting before the final renderer (last processor in the chain).

Review Comment:
   The injection itself is inert for CloudWatch: every global-chain logger is stdout/pipe-backed, so `relative_path_from_logger()` returns `None` and the processor early-returns. Actual streaming uses the file-backed task logger from `logging_processors()`, which loads `remote.processors` lazily — also after `dictConfig`.
   
   ```suggestion
       # dictConfig has now run, so it is safe to build the watchtower handler. Re-inject the remote
       # processors into the global structlog chain (before the final renderer) for parity with the old
       # extra_processors layout. Task-log streaming itself does not rely on this: it uses the
       # file-backed logger built from logging_processors(), which loads remote.processors lazily.
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3565598238


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2405,9 +2422,30 @@ def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLog
 
     with _remote_logging_conn(client):
         processors = logging_processors(json_output=json_logs)
+
     logger = structlog.wrap_logger(underlying_logger, processors=processors, logger_name="task").bind()
 
-    return logger, log_file_descriptor
+    try:
+        yield logger
+    finally:
+        # Flush and close the remote handler now — AFTER the supervisor has
+        # drained all task log messages from the subprocess pipe (i.e. after
+        # process.wait() has returned).
+        #
+        # Without this, the only thing that ever closes the handler is
+        # Python's logging.shutdown() at process exit, which fires after
+        # supervise_task() returns. Any messages still queued in the handler
+        # at that point are silently dropped. For example, the AWS CloudWatch
+        # logger will emit:
+
+        # WatchtowerWarning: "Received message after logging system shutdown"
+        remote_handler = load_remote_log_handler()
+        if remote_handler is not None:

Review Comment:
   Applied suggestion. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4993453395

   Addressed all outstanding review comments:
   
   - Reverted unrelated changes in `_handle_request` (GetXComCount ordering and send_msg parameter style)
   - Restored the dropped TODO comment in `_configure_logging`
   - Used walrus operator for `load_remote_log_handler()` check
   - Added `hasattr(handler, "close")` guard in `_close_remote_log_handler` to avoid mypy error after reverting the RemoteLogIO protocol change
   - Reverted all provider and shared/logging changes per @jason810496 guidance, this PR now touches only supervisor.py and log.py


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5004218664

   > Can the provider related changes be split into a dedicated PR?
   
   Done!
   All provider and shared/logging changes were already reverted in the latest push per @jason810496's guidance. This PR now touches only task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. The provider-side fix is being handled separately in #68779.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-6055086571

   I'll take a look today 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4444809696

   > I would have expected some new or more updated test cases to ensure this doesn't regress again? Do you not think that's required here @korex-f ?
   
   Good point, I have added tests to test_cloudwatch_task_handler.py covering the self-healing handler property (verifies a new handler is created when shutting_down=True) and the dynamic self.handler access in the processors closure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3363895769


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2189,11 +2255,30 @@ def ensure_secrets_backend_loaded() -> list[BaseSecretsBackend]:
     return ensure_secrets_loaded(default_backends=fallback_backends)
 
 
-def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLogger, BinaryIO | TextIO]:
+def _close_remote_log_handler(handler: RemoteLogIO) -> None:
+    """
+    Close the remote log handler explicitly after all task log messages have been drained from the subprocess pipe.
+
+    Called after process.wait() returns, before process exit triggers
+    logging.shutdown(). This ensures Watchtower's internal batch queue
+    is flushed to CloudWatch before the process tears down.
+    """
+    with contextlib.suppress(Exception):

Review Comment:
   Logged it, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3363892957


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -467,7 +468,8 @@ def exit(n: int) -> NoReturn:
         except Exception as e:
             with suppress(Exception):
                 print(
-                    f"--- Last chance exception handler failed --- {repr(str(e))}\n", file=last_chance_stderr
+                    f"--- Last chance exception handler failed --- {repr(str(e))}\n",
+                    file=last_chance_stderr,

Review Comment:
   Done
   



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -628,13 +630,22 @@ def start(
                     # execv replaces the process -- unreachable on success
                 else:
                     # Run the child entrypoint
-                    _fork_main(child_requests, child_stdout, child_stderr, child_logs.fileno(), target)
+                    _fork_main(
+                        child_requests,
+                        child_stdout,
+                        child_stderr,
+                        child_logs.fileno(),
+                        target,

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3363891483


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -39,8 +41,6 @@
 from airflow.utils.log.logging_mixin import LoggingMixin
 
 if TYPE_CHECKING:
-    import structlog.typing

Review Comment:
   Done that, thanks
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3328322732


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,38 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    _handler_creating: bool = attrs.field(default=False, init=False, repr=False)
+
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        if self._handler_creating:
+            # Re-entrant call during handler creation, some libraries log internally
+            # during initialization, which triggers this process again before
+            # handler is fully constructed.
+            if self._handler is not None and isinstance(self._handler, watchtower.CloudWatchLogHandler):
+                return self._handler
+            raise structlog.DropEvent()
+        if self._handler is None or self._handler.shutting_down:
+            self._handler_creating = True
+            try:
+                self._handler = self._create_handler()
+            finally:
+                self._handler_creating = False
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly initialize the handler before the closure is created.
+        # Without this, the first log emission would trigger handler creation,
+        # which itself logs, causing an infinite loop.
+        self._handler

Review Comment:
   > @korex-f thoughts on this one?
   
   Agreed with ferruzzi's observation, and it's prompted me to reconsider the whole self-healing property approach. 
   After working through the re-entrancy issues this introduced, I think your earlier question was the right one, the self-healing handler property may be over-engineering given Bug 1 already prevents the handler from being created before dictConfig runs.
   I'm considering reverting handler back to cached_property and keeping only the processors closure fix (accessing self.handler dynamically instead of a captured _handler variable). That eliminates the recursion complexity while still fixing the closure-pinning issue.
   
   Does that seem like the right scope to you???



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "sarvesh371 (via GitHub)" <gi...@apache.org>.
sarvesh371 commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4681585940

   Hey, running into this exact issue on Airflow 3.2.2 with KubernetesExecutor — task pods emit WatchtowerWarning: Received message after logging system shutdown and logs never make it to CloudWatch.
   
   Any idea on ETA for merge and which provider version will ship the fix? Also is there any interim workaround while we wait?
   
   Happy to test if there's a pre-release available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3707284685


##########
task-sdk/tests/task_sdk/test_log.py:
##########
@@ -0,0 +1,166 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import structlog
+import structlog.testing
+from uuid6 import uuid7
+
+from airflow.sdk import log as sdk_log
+
+
+def _make_ti():
+    ti = mock.MagicMock()
+    ti.id = uuid7()
+    return ti
+
+
+def _make_logger():
+    """Build a FilteringBoundLogger-like object exposing ``_logger``."""
+    logger = mock.MagicMock()
+    logger._logger = mock.MagicMock()
+    return logger
+
+
+class TestUploadToRemote:

Review Comment:
   The three test_warns_* tests in this class fail at HEAD, see the Task SDK job: https://github.com/apache/airflow/actions/runs/30734933337/job/91463012820. They assert warning events (remote_log_handler_unavailable, remote_log_path_resolution_failed, remote_log_upload_failed) that only existed in the earlier revision of this PR; after the scope-down nothing emits them, and upload_to_remote doesn't catch upload errors, so test_warns_when_upload_fails fails with the raw RuntimeError. The Glue test you mentioned is a different job, this failure comes from this file. I'd drop these three and keep the two test_silent_* tests plus the dictConfig regression test, which do match current behavior.



##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -119,8 +119,14 @@ def configure_logging(
     if mask_secrets:
         extra_processors += (mask_logs,)
 
-    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
-        extra_processors += remote_processors
+    # NOTE: Do NOT call getattr(remote, "processors") here.
+    # Accessing remote.processors triggers creation of the remote handler
+    # via a cached_property. The configure_logging() call below runs dictConfig() internally,

Review Comment:
   The "Done. Thanks" replies from Aug 2 don't appear to be in the branch, the last commit (3217a41) predates @jason810496's comments: this line still mentions the cached_property, and the two assert-style nits in test_log.py are unchanged. Could you push the commit with those changes? The threads are marked resolved, so this is easy to miss otherwise.



##########
task-sdk/tests/task_sdk/test_log.py:
##########
@@ -0,0 +1,166 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import structlog
+import structlog.testing
+from uuid6 import uuid7
+
+from airflow.sdk import log as sdk_log
+
+
+def _make_ti():
+    ti = mock.MagicMock()
+    ti.id = uuid7()
+    return ti
+
+
+def _make_logger():
+    """Build a FilteringBoundLogger-like object exposing ``_logger``."""
+    logger = mock.MagicMock()
+    logger._logger = mock.MagicMock()
+    return logger
+
+
+class TestUploadToRemote:
+    def test_warns_when_handler_unavailable(self):
+        ti = _make_ti()
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=None),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_handler_unavailable"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+
+    def test_warns_when_path_resolution_fails(self):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        boom = RuntimeError("cannot resolve path")
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", side_effect=boom),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_path_resolution_failed"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+        assert events[0]["exc_info"] is boom
+        handler.upload.assert_not_called()
+
+    def test_warns_when_upload_fails(self, tmp_path):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        boom = RuntimeError("s3 unreachable")
+        handler.upload.side_effect = boom
+        relative = tmp_path / "dag_id" / "run_id" / "task.log"
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", return_value=relative),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_upload_failed"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+        assert events[0]["log_relative_path"] == relative.as_posix()
+        assert events[0]["exc_info"] is boom
+        handler.upload.assert_called_once_with(relative.as_posix(), ti)
+
+    def test_silent_when_relative_path_is_none(self):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", return_value=None),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        assert captured == []
+        handler.upload.assert_not_called()
+
+    def test_silent_on_success(self, tmp_path):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        relative = tmp_path / "dag_id" / "run_id" / "task.log"
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", return_value=relative),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        assert captured == []
+        handler.upload.assert_called_once_with(relative.as_posix(), ti)
+
+
+class TestConfigureLogging:
+    def test_remote_processors_injected_after_dictconfig(self):
+        """
+        Regression test: remote processor injection must happen AFTER dictConfig() runs.
+
+        dictConfig()'s non-incremental reset closes every handler in
+        logging._handlerList. If the remote handler is built before dictConfig
+        runs, it is closed before any task log is emitted and silently drops
+        all records.
+        """
+        import airflow.sdk._shared.logging as shared_logging
+
+        call_order = []
+
+        mock_handler = mock.MagicMock()
+        mock_handler.processors = (mock.MagicMock(),)
+
+        def track_load_remote():
+            call_order.append("load_remote_log_handler")
+            return mock_handler
+
+        original_inner = shared_logging.configure_logging
+
+        def track_inner_configure(*args, **kwargs):
+            call_order.append("dictConfig")
+            return original_inner(*args, **kwargs)
+
+        # configure_logging is @cache decorated — clear it so the test actually runs the function
+        sdk_log.configure_logging.cache_clear()

Review Comment:
   This test leaves global state behind: the configure_logging cache is cleared but never restored, and the real structlog.configure call inside puts mock_handler's MagicMock processor into the live global chain, where it stays for every later test in the process until something reconfigures logging. Restoring the previous processors in a finally (the way captured_logs in conftest.py does) and clearing the cache again on the way out would keep it isolated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3699843290


##########
task-sdk/tests/task_sdk/test_log.py:
##########
@@ -0,0 +1,166 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import structlog
+import structlog.testing
+from uuid6 import uuid7
+
+from airflow.sdk import log as sdk_log
+
+
+def _make_ti():
+    ti = mock.MagicMock()
+    ti.id = uuid7()
+    return ti
+
+
+def _make_logger():
+    """Build a FilteringBoundLogger-like object exposing ``_logger``."""
+    logger = mock.MagicMock()
+    logger._logger = mock.MagicMock()
+    return logger
+
+
+class TestUploadToRemote:
+    def test_warns_when_handler_unavailable(self):
+        ti = _make_ti()
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=None),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_handler_unavailable"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+
+    def test_warns_when_path_resolution_fails(self):
+        ti = _make_ti()
+        handler = mock.MagicMock()
+        boom = RuntimeError("cannot resolve path")
+        with (
+            mock.patch.object(sdk_log, "load_remote_log_handler", return_value=handler),
+            mock.patch.object(sdk_log, "relative_path_from_logger", side_effect=boom),
+            structlog.testing.capture_logs() as captured,
+        ):
+            sdk_log.upload_to_remote(_make_logger(), ti)
+
+        events = [e for e in captured if e["event"] == "remote_log_path_resolution_failed"]
+        assert len(events) == 1
+        assert events[0]["log_level"] == "warning"
+        assert events[0]["ti_id"] == str(ti.id)
+        assert events[0]["exc_info"] is boom

Review Comment:
   Done. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3699842323


##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -119,8 +119,14 @@ def configure_logging(
     if mask_secrets:
         extra_processors += (mask_logs,)
 
-    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
-        extra_processors += remote_processors
+    # NOTE: Do NOT call getattr(remote, "processors") here.
+    # Accessing remote.processors triggers creation of the remote handler
+    # via a cached_property. The configure_logging() call below runs dictConfig() internally,

Review Comment:
   Done. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5131038882

   > Hi @korex-f,
   > 
   > Thanks for follow-up on this. We need to do the following items before merge.
   > 
   >     * Add unit test to exercise the patch itself
   > 
   >     * Fix the CI failure
   > 
   >     * Rephrase the PR title and the description
   > 
   > 
   > Thanks.
   
   The CI failure in test_inject_parent_job_info_with_resume_on_retry is unrelated to this PR,  it's a Glue operator test making a real network call that times out after 60s. Our PR only touches log.py. The test was last modified in #64513 (OpenLineage injection) and appears to be a pre-existing issue. Happy to re-trigger CI to confirm it's a flake if that helps.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3624668016


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -883,11 +856,7 @@ def _serialize_response(self, msg: BaseModel | ErrorResponse, **dump_opts) -> di
         return msg.model_dump(**dump_opts)
 
     def send_msg(
-        self,
-        msg: BaseModel | None,
-        request_id: int,
-        error: ErrorResponse | None = None,
-        **dump_opts,
+        self, msg: BaseModel | None, request_id: int, error: ErrorResponse | None = None, **dump_opts
     ):

Review Comment:
   ^^ Same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3237116710


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2189,11 +2190,36 @@ def ensure_secrets_backend_loaded() -> list[BaseSecretsBackend]:
     return ensure_secrets_loaded(default_backends=fallback_backends)
 
 
-def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLogger, BinaryIO | TextIO]:
+def _close_remote_log_handler(handler: RemoteLogIO) -> None:
+    """
+    Close the remote log handler explicitly after all task log messages have been drained, before process exit triggers logging.shutdown().
+
+    This prevents WatchtowerWarning: "Received message after logging system shutdown" which causes CloudWatch log streams to be silently truncated.
+    """
+    import logging as _logging
+
+    # If the handler is also a stdlib logging.Handler, deregister it from
+    # the logging hierarchy so logging.shutdown() at process exit cannot
+    # double-close it.
+    if isinstance(handler, _logging.Handler):
+        root = _logging.getLogger()

Review Comment:
   You're right, the isinstance guard is dead code. None of the current  RemoteLogIO implementations inherit from logging.Handler. With Bug 1 fixed the watchtower handler is built after dictConfig runs so it's never 
   registered in the hierarchy in the first place. I have removed the dead block and simplified _close_remote_log_handler to just handler.close().



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3237116710


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2189,11 +2190,36 @@ def ensure_secrets_backend_loaded() -> list[BaseSecretsBackend]:
     return ensure_secrets_loaded(default_backends=fallback_backends)
 
 
-def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLogger, BinaryIO | TextIO]:
+def _close_remote_log_handler(handler: RemoteLogIO) -> None:
+    """
+    Close the remote log handler explicitly after all task log messages have been drained, before process exit triggers logging.shutdown().
+
+    This prevents WatchtowerWarning: "Received message after logging system shutdown" which causes CloudWatch log streams to be silently truncated.
+    """
+    import logging as _logging
+
+    # If the handler is also a stdlib logging.Handler, deregister it from
+    # the logging hierarchy so logging.shutdown() at process exit cannot
+    # double-close it.
+    if isinstance(handler, _logging.Handler):
+        root = _logging.getLogger()

Review Comment:
   You're right, the isinstance guard is dead code. None of the current 
   RemoteLogIO implementations inherit from logging.Handler. With Bug 1 fixed 
   the watchtower handler is built after dictConfig runs so it's never 
   registered in the hierarchy in the first place. I have removed the dead block 
   and simplified _close_remote_log_handler to just handler.close().



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3237146003


##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2209,9 +2235,29 @@ def _configure_logging(log_path: str, client: Client) -> tuple[FilteringBoundLog
 
     with _remote_logging_conn(client):
         processors = logging_processors(json_output=json_logs)
+
     logger = structlog.wrap_logger(underlying_logger, processors=processors, logger_name="task").bind()
 
-    return logger, log_file_descriptor
+    try:
+        yield logger, log_file_descriptor
+    finally:
+        # Flush and close the remote handler now — AFTER the supervisor has
+        # drained all task log messages from the subprocess pipe (i.e. after
+        # process.wait() has returned).
+        #
+        # Without this, the only thing that ever closes the handler is
+        # Python's logging.shutdown() at process exit, which fires after
+        # supervise_task() returns. Any messages still queued in the handler
+        # at that point are silently dropped, producing:
+        #
+        # WatchtowerWarning: "Received message after logging system shutdown"

Review Comment:
   Thanks. I have made those comments provider-agnostic and use CloudWatch as an example rather than stating it as the only case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3698259238


##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -119,8 +119,14 @@ def configure_logging(
     if mask_secrets:
         extra_processors += (mask_logs,)
 
-    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
-        extra_processors += remote_processors
+    # NOTE: Do NOT call getattr(remote, "processors") here.
+    # Accessing remote.processors triggers creation of the remote handler
+    # via a cached_property. The configure_logging() call below runs dictConfig() internally,

Review Comment:
   We don't need to mention the `cached_property` workaround anymore. Since this PR address the root case.
   
   ```suggestion
       # The configure_logging() call below runs dictConfig() internally,
   ```



##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -119,8 +119,14 @@ def configure_logging(
     if mask_secrets:
         extra_processors += (mask_logs,)
 
-    if (remote := load_remote_log_handler()) and (remote_processors := getattr(remote, "processors")):
-        extra_processors += remote_processors
+    # NOTE: Do NOT call getattr(remote, "processors") here.
+    # Accessing remote.processors triggers creation of the remote handler
+    # via a cached_property. The configure_logging() call below runs dictConfig() internally,

Review Comment:
   We don't need to mention the `cached_property` workaround anymore. Since this PR address the root cause.
   
   ```suggestion
       # The configure_logging() call below runs dictConfig() internally,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4668162677

   Thanks for the review, @ashb. I have addressed the requested changes, pushed the fixes, and re-requested review. Please do take another look when you have the chance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4995488505

   Can the provider related changes be split into a dedicated PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "chris-stetter (via GitHub)" <gi...@apache.org>.
chris-stetter commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-4419937257

   I can confirm that this works in our ECS environment. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3344849352


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,38 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    _handler_creating: bool = attrs.field(default=False, init=False, repr=False)
+
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        if self._handler_creating:
+            # Re-entrant call during handler creation, some libraries log internally
+            # during initialization, which triggers this process again before
+            # handler is fully constructed.
+            if self._handler is not None and isinstance(self._handler, watchtower.CloudWatchLogHandler):
+                return self._handler
+            raise structlog.DropEvent()
+        if self._handler is None or self._handler.shutting_down:
+            self._handler_creating = True
+            try:
+                self._handler = self._create_handler()
+            finally:
+                self._handler_creating = False
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly initialize the handler before the closure is created.
+        # Without this, the first log emission would trigger handler creation,
+        # which itself logs, causing an infinite loop.
+        self._handler

Review Comment:
   I think so, yeah.   As long as that still fixes the log issue, then that sounds like the cleaner approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix: CloudWatch/Watchtower logs dropped in Task SDK due to handler lifetime bugs [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #66633:
URL: https://github.com/apache/airflow/pull/66633#discussion_r3320767602


##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -118,16 +119,38 @@ def handler(self) -> watchtower.CloudWatchLogHandler:
             json_serialize_default=_json_serialize or json_serialize_legacy,
         )
 
+    _handler_creating: bool = attrs.field(default=False, init=False, repr=False)
+
+    @property
+    def handler(self) -> watchtower.CloudWatchLogHandler:
+        if self._handler_creating:
+            # Re-entrant call during handler creation, some libraries log internally
+            # during initialization, which triggers this process again before
+            # handler is fully constructed.
+            if self._handler is not None and isinstance(self._handler, watchtower.CloudWatchLogHandler):
+                return self._handler
+            raise structlog.DropEvent()
+        if self._handler is None or self._handler.shutting_down:
+            self._handler_creating = True
+            try:
+                self._handler = self._create_handler()
+            finally:
+                self._handler_creating = False
+        return self._handler
+
     @cached_property
     def processors(self) -> tuple[structlog.typing.Processor, ...]:
         from logging import getLogRecordFactory
 
         import structlog.stdlib
 
         logRecordFactory = getLogRecordFactory()
-        # The handler MUST be initted here, before the processor is actually used to log anything.
-        # Otherwise, logging that occurs during the creation of the handler can create infinite loops.
-        _handler = self.handler
+        # Eagerly initialize the handler before the closure is created.
+        # Without this, the first log emission would trigger handler creation,
+        # which itself logs, causing an infinite loop.
+        self._handler

Review Comment:
   @korex-f  thoughts on this one?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix remote processor injection happening before dictConfig runs in configure_logging [airflow]

Posted by "korex-f (via GitHub)" <gi...@apache.org>.
korex-f commented on PR #66633:
URL: https://github.com/apache/airflow/pull/66633#issuecomment-5792030069

   Thanks @potiuk. I have added the assertion so the test now confirms that a remote handler without `processors` leaves the returned chain with only the standard processors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org