[PR] Fix remote log handlers crashing triggerer logging thread on concurrent close [airflow]

["ZhaoMJ (via GitHub)" <gi...@apache.org>]

ZhaoMJ opened a new pull request, #70479:
URL: https://github.com/apache/airflow/pull/70479

   Remote task log handlers (S3, GCS, WASB, OSS, Elasticsearch, HDFS) reclaim disk on upload by calling `shutil.rmtree(os.path.dirname(local_loc))` when `delete_local_copy` is enabled — removing the log file's **parent directory** rather than the file itself.
   
   The triggerer runs many trigger log handlers concurrently in a single process, and handler close is **not atomic**: the main thread's `logging.shutdown()` and the `QueueListener` monitor thread draining a `trigger_end` record can both call `close()` → `upload()` on the same handler. Two `rmtree` calls then run against the same directory — the first removes the tree, the second walks a path that just vanished and raises `FileNotFoundError`. Because it is raised on the logging monitor thread, it is unhandled and **kills the listener thread**, silently stopping all trigger log delivery for the remainder of the process's life.
   
   Observed traceback (provider `amazon` 9.2.0, Airflow 2.11.0):
   
   ```
   Exception in thread Thread-1 (_monitor):
     File ".../logging/handlers.py", line 1598, in _monitor
       self.handle(record)
     File ".../airflow/utils/log/trigger_handler.py", line 113, in close_one
       h.close()
     File ".../airflow/providers/amazon/aws/log/s3_task_handler.py", line 102, in close
       shutil.rmtree(os.path.dirname(local_loc))
     ...
   FileNotFoundError: [Errno 2] No such file or directory: '.../task_id=.../map_index=0'
   ```
   
   ### Fix
   
   Delete only the uploaded file (`unlink(missing_ok=True)`) and then prune now-empty parent directories, stopping at the first non-empty parent and at `base_log_folder`. This is the same pattern the OpenSearch handler already uses (added in #64364). It:
   
   - is **idempotent** under concurrent/double close (`missing_ok=True` + `contextlib.suppress(OSError)`),
   - never removes a directory a concurrent sibling handler is still using (`if any(parent.iterdir()): break`),
   - still reclaims disk (the log bytes are removed; only empty dirs are pruned).
   
   Applied to all six affected handlers (`opensearch` already had it).
   
   ### Tests
   
   Added a regression test for the S3 handler asserting the delete is idempotent and that a shared parent directory with a sibling log is preserved.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   <!-- Generated-by: Claude Code following the guidelines -->
   Generated-by: Claude Code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix remote log handlers crashing triggerer logging thread on concurrent close [airflow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #70479:
URL: https://github.com/apache/airflow/pull/70479#issuecomment-5642153550

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix remote log handlers crashing triggerer logging thread on concurrent close [airflow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] closed pull request #70479: Fix remote log handlers crashing triggerer logging thread on concurrent close
URL: https://github.com/apache/airflow/pull/70479


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org