You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/06/18 12:20:26 UTC
[PR] Propagate resolved task log level to language SDK runtimes [airflow]
jason810496 opened a new pull request, #68712:
URL: https://github.com/apache/airflow/pull/68712
- related: https://github.com/apache/airflow/pull/68696#discussion_r3433102278
## Why
Pointed out in above comment, we should propagate the `logging_level` info from Supervisor (Task-SDK world) to Lang-SDK runtime to respect the Airflow side `conf`. So that no matter user configure the `[logging/logging_level]` setting using `airflow.cfg` or secret backend, the Lang-SDK should retrieve the same conf.
## How
Here're the approach we came up with:
1. **(Current approach)** Passing as `AIRFLOW__LOGGING__LOGGING_LEVEL` env, each Lang-SDK will consume by `os.environ` themself.
2. Passing by CLI flag like `java -classpath ... --comms=... --logs=... --logging-level=...` CLI. Cons: We will introduce a new CLI flag for all the further `conf` propagation. (There isn't `--comms` or `--logs` equivalent Airflow env, so having the existing `--comms, --logs` as-is is a valid direction)
3. Passing by `StartUpDetails.logging_level`. Cons: Caught by @uranusjr that the Task subprocess had already started producing logs (e.g. logging for connecting to the coordinator) before the Task subprocess got the `StartUpDetails` msgpack, which causes the wrong logging level at the Task subprocess startup time.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes, with help of Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Propagate resolved task log level to language SDK runtimes [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #68712:
URL: https://github.com/apache/airflow/pull/68712#discussion_r3435954301
##########
task-sdk/src/airflow/sdk/coordinators/_subprocess.py:
##########
@@ -301,6 +302,13 @@ def start( # type: ignore[override]
],
stdout=stdout_w.fileno(),
stderr=stderr_w.fileno(),
+ # A language SDK runtime cannot read Airflow's config, so propagate the
+ # resolved task log level via the environment at launch. StartupDetails
+ # arrives too late, the logs might already be produced by then.
+ env={
+ **os.environ,
+ "AIRFLOW__LOGGING__LOGGING_LEVEL": conf.get("logging", "logging_level", fallback="INFO"),
Review Comment:
If we are doing this, we should also propagate the (probably badly names) namespace logger levels too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Propagate resolved task log level to language SDK runtimes [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #68712:
URL: https://github.com/apache/airflow/pull/68712#discussion_r3438002085
##########
task-sdk/src/airflow/sdk/coordinators/_subprocess.py:
##########
@@ -301,6 +302,13 @@ def start( # type: ignore[override]
],
stdout=stdout_w.fileno(),
stderr=stderr_w.fileno(),
+ # A language SDK runtime cannot read Airflow's config, so propagate the
+ # resolved task log level via the environment at launch. StartupDetails
+ # arrives too late, the logs might already be produced by then.
+ env={
+ **os.environ,
+ "AIRFLOW__LOGGING__LOGGING_LEVEL": conf.get("logging", "logging_level", fallback="INFO"),
Review Comment:
Yeah good point, we should.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Propagate resolved task log level to language SDK runtimes [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #68712:
URL: https://github.com/apache/airflow/pull/68712#issuecomment-4744877972
We should also mention this in sdk contribution doc. Each sdk needs to handle this on their own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Propagate resolved task log level to language SDK runtimes [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #68712:
URL: https://github.com/apache/airflow/pull/68712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Propagate resolved task log level to language SDK runtimes [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #68712:
URL: https://github.com/apache/airflow/pull/68712#issuecomment-4763390315
I’m moving the documentation addition to #68603 so we can also add agent instructions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org