You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shahar1 (via GitHub)" <gi...@apache.org> on 2026/10/17 17:48:28 UTC
[PR] Prevent Dag processor crashes with Google Cloud Logging when its log directory is outside `base_log_folder` [airflow]
shahar1 opened a new pull request, #74926:
URL: https://github.com/apache/airflow/pull/74926
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
related: #68240
<!-- HUMAN SUMMARY - AGENTS DO NOT MODIFY -->
When the Dag processor log directory is outside `base_log_folder`, Google Cloud Logging raises `ValueError` and crashes before parsing the DAG.
This catches the error and keeps those lines local; verified on Airflow 3.3.2, where the Dag parsed and its triggered run succeeded.
AWS CloudWatch has the same issue, but it isn’t fixed here.
<!-- END OF HUMAN SUMMARY -->
<details><summary>AI Summary</summary>
With Google Cloud Logging remote logging, the Dag processor crashes on its first log line when `[logging] dag_processor_child_process_log_directory` points outside `[logging] base_log_folder`:
```
dag-processor | [error] Exception when executing DagProcessorJob
File ".../cloud_logging_task_handler.py", line 180, in proc
if not logger or not relative_path_from_logger(logger):
ValueError: '/tmp/dagproc-logs/2026-10-17/dags-folder/repro_68293.py.log' is not in the subpath of '/opt/airflow/logs' ...
```
`CloudLoggingRemoteLogIO.processors` calls `relative_path_from_logger`, which uses `Path.relative_to` and raises `ValueError` for files outside `base_log_folder`. Nothing on the way up catches it, so the exception ends the Dag processor job and no Dag is parsed. `upload_to_remote` already skips such files. The processor now does the same: it uses a wrapper that returns `None` on `ValueError`, so these lines stay in the local log file and are not sent to Cloud Logging.
The guard sits where `relative_path_from_logger` is imported, not on the processor's early-return line, so this merges cleanly with #74923, which changes that line.
Verified on the official Airflow 3.3.2 image writing to Cloud Logging, with the Dag processor log directory outside `base_log_folder`:
- **Provider built from main:** the Dag processor job fails with the `ValueError` above and no Dag is parsed.
- **This PR:** the Dag processor keeps running, the Dag is parsed, and a triggered run succeeds with its task logs in Cloud Logging.
This may also be what #68240 describes as a Dag processor in `CrashLoopBackOff` "on every log emit". That report attributed it to an IAM error in `transport.send()`, which the default transport cannot raise.
Not covered here: the Amazon CloudWatch remote log IO (`cloudwatch_task_handler.py`) calls `relative_path_from_logger` the same way and has the same crash.
</details>
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent Dag processor crashes with Google Cloud Logging when its log directory is outside `base_log_folder` [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #74926:
URL: https://github.com/apache/airflow/pull/74926#discussion_r4303824508
##########
providers/google/src/airflow/providers/google/cloud/log/cloud_logging_task_handler.py:
##########
@@ -167,7 +167,15 @@ def transport(self) -> Transport:
def processors(self) -> tuple[structlog.typing.Processor, ...]:
import structlog.stdlib
- from airflow.sdk.log import relative_path_from_logger
+ from airflow.sdk.log import relative_path_from_logger as _relative_path_from_logger
+
+ def relative_path_from_logger(logger) -> Path | None:
+ try:
+ return _relative_path_from_logger(logger)
+ except ValueError:
+ # Skip log files outside ``base_log_folder``, such as a Dag processor log directory
+ # configured elsewhere, as ``upload_to_remote`` does.
Review Comment:
Perhaps you could add a debug log here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent Dag processor crashes with Google Cloud Logging when its log directory is outside `base_log_folder` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #74926:
URL: https://github.com/apache/airflow/pull/74926#discussion_r4367960953
##########
providers/google/src/airflow/providers/google/cloud/log/cloud_logging_task_handler.py:
##########
@@ -167,7 +167,15 @@ def transport(self) -> Transport:
def processors(self) -> tuple[structlog.typing.Processor, ...]:
import structlog.stdlib
- from airflow.sdk.log import relative_path_from_logger
+ from airflow.sdk.log import relative_path_from_logger as _relative_path_from_logger
+
+ def relative_path_from_logger(logger) -> Path | None:
Review Comment:
Non-blocking follow-up: the real source of the problem is `airflow.sdk.log.relative_path_from_logger`, which lets `Path.relative_to` raise. Making it return `None` for paths outside `base_log_folder` would fix this for every caller. This guard would still be needed for older Task SDK versions, but new remote-log IOs wouldn't have to rediscover the problem.
---
Drafted-by: Claude Opus 5.5; reviewed by `potiuk` before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent Dag processor crashes with Google Cloud Logging when its log directory is outside `base_log_folder` [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #74926:
URL: https://github.com/apache/airflow/pull/74926#discussion_r4306437306
##########
providers/google/src/airflow/providers/google/cloud/log/cloud_logging_task_handler.py:
##########
@@ -167,7 +167,15 @@ def transport(self) -> Transport:
def processors(self) -> tuple[structlog.typing.Processor, ...]:
import structlog.stdlib
- from airflow.sdk.log import relative_path_from_logger
+ from airflow.sdk.log import relative_path_from_logger as _relative_path_from_logger
+
+ def relative_path_from_logger(logger) -> Path | None:
+ try:
+ return _relative_path_from_logger(logger)
+ except ValueError:
+ # Skip log files outside ``base_log_folder``, such as a Dag processor log directory
+ # configured elsewhere, as ``upload_to_remote`` does.
Review Comment:
This processor runs in the Dag processor manager for every log line forwarded from every Dag file parse. A debug line here fires once per log line, so turning on `DEBUG` would roughly double the manager's own output with the same message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent Dag processor crashes with Google Cloud Logging when its log directory is outside `base_log_folder` [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #74926:
URL: https://github.com/apache/airflow/pull/74926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org