You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "plmnguyen (via GitHub)" <gi...@apache.org> on 2024/01/31 17:01:02 UTC
[I] Allow Dag Processor Manager to log to stdout [airflow]
plmnguyen opened a new issue, #37122:
URL: https://github.com/apache/airflow/issues/37122
### Description
On Airflow 2.8.1, the DAG processor manager no longer logs out to stdout (whereas it did on 2.6.2).
### Use case/motivation
We currently have a configuration option for the DAG processor to log to stdout via `AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET`. A similar configuration `AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_TARGET` could achieve that.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow Dag Processor Manager to log to stdout [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37122:
URL: https://github.com/apache/airflow/issues/37122#issuecomment-1934125465
> thoughts ?
This is a valid option if someone wants it, all it needs is someone to PR it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow Dag Processor Manager to log to stdout [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37122:
URL: https://github.com/apache/airflow/issues/37122#issuecomment-1919526365
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow Dag Processor Manager to log to stdout [airflow]
Posted by "Owen-CH-Leung (via GitHub)" <gi...@apache.org>.
Owen-CH-Leung commented on issue #37122:
URL: https://github.com/apache/airflow/issues/37122#issuecomment-1934151048
> > thoughts ?
>
> This is a valid option if someone wants it, all it needs is someone to PR it.
Ok I'm on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow Dag Processor Manager to log to stdout [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #37122: Allow Dag Processor Manager to log to stdout
URL: https://github.com/apache/airflow/issues/37122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow Dag Processor Manager to log to stdout [airflow]
Posted by "Owen-CH-Leung (via GitHub)" <gi...@apache.org>.
Owen-CH-Leung commented on issue #37122:
URL: https://github.com/apache/airflow/issues/37122#issuecomment-1933976278
afaik, the DAG processor manager runs in a spawned process and by default output its log to a destinated file defined by `dag_processor_manager_log_location`. I think this behaviour also applies to `v2.6.2`
I suggest adding an extra variable in the config like `dag_processor_manager_log_stdout` , which if being set to `True` (default is `False`) , extend the log handler to also include logging to stdout. That way, the log from the spawned process can also be directed to the stdout of airflow scheduler like below :
<img width="1399" alt="image" src="https://github.com/apache/airflow/assets/43698890/645fe926-ed81-4766-80c0-e0a3fa673155">
thoughts ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow Dag Processor Manager to log to stdout [airflow]
Posted by "stevenschaerer (via GitHub)" <gi...@apache.org>.
stevenschaerer commented on issue #37122:
URL: https://github.com/apache/airflow/issues/37122#issuecomment-1925885502
Happy to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow Dag Processor Manager to log to stdout [airflow]
Posted by "jeff-cook (via GitHub)" <gi...@apache.org>.
jeff-cook commented on issue #37122:
URL: https://github.com/apache/airflow/issues/37122#issuecomment-3757380282
I found the config doc for this https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-processor-log-target
And set
```
env:
- name: AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET
value: stdout
```
However in `apache/airflow:3.0.2` it is not sending to stdout.
I'm running the current helm chart version `1.18.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org