You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mert-kirpici (via GitHub)" <gi...@apache.org> on 2024/02/08 22:24:38 UTC
[I] Add configuration option to rotate and/or push scheduler logs to remote [airflow]
mert-kirpici opened a new issue, #37266:
URL: https://github.com/apache/airflow/issues/37266
### Description
I have remote logging enabled, the task logs are nicely being pushed to S3. However the scheduler logs keep growing on disk and afaik there is no way to configure airflow to push them to remote also, or set up some sort of retention period to rotate them without relying on an external mechanism like logrotated or crond. I believe this would be a useful feature.
### Use case/motivation
- A new configuration option `scheduler.logging_retention_period` to keep set up the retention period for the scheduler logs
AND/OR
(
- The `logging.remote_logging` option could also upload scheduler logs to the remote location
OR
- A new configuration option `scheduler.remote_logging` to be able to push scheduler logs to remote seperately
)
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add configuration option to rotate and/or push scheduler logs to remote [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37266:
URL: https://github.com/apache/airflow/issues/37266#issuecomment-1935091821
You can already do that by implementing cusom handlers via https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html.
We are using standard python logging functionality and even added a way to configure loggers via dictionary.
If you want to log things remotely, you can find python logging handler that does and plug them in that advanced configuration - no problem with that.
I see no particular reason why Airflow would have to add additional layer of abstraction there and be opinionated about your logging choice where standard Python logging framework is fully extensible, has a number of handlers implemented (including Rotating handlers).
There are a number of handlers implemented already that can do any kind of logging remotely. Pretty much every cloud storage provider or logging system provider comes with a python logging handler that you can just plug-in (and custom configuration of ours allows you to plug them in in whatever way you think makes sense - using standard hierarchy of loggers based on python packages.
The only reason we have `logging task handlers` and `remote logging` for tasks is that they need to be integrated in Airflow UI and by those remote handlers we can not only push stuff to S3 but also pull it for the UI purposse
I do not think we should add something extra here, rather than that I'd focus on adding Open Telemetry logging integration which we already agreed to as an AIP (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow) . This is much better abstractiion (and one that we do not have to define and has a chance to be well standardized) and out-of-the-box has support for pretty much anything.
Converting it into a discussion if more is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add configuration option to rotate and/or push scheduler logs to remote [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #37266: Add configuration option to rotate and/or push scheduler logs to remote
URL: https://github.com/apache/airflow/issues/37266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add configuration option to rotate and/or push scheduler logs to remote [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37266:
URL: https://github.com/apache/airflow/issues/37266#issuecomment-1935029374
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org