You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/29 10:20:32 UTC
[GitHub] [airflow] internetcoffeephone opened a new issue #18604: Allow multiple simultaneous remote logging methods
internetcoffeephone opened a new issue #18604:
URL: https://github.com/apache/airflow/issues/18604
### Description
Allow multiple simultaneous remote logging methods.
### Use case/motivation
Currently, Airflow only allows for one remote logging option at a time. This is done in the block starting here: https://github.com/apache/airflow/blob/c4fd84accd143977cba57e4daf6daef2af2ff457/airflow/config_templates/airflow_local_settings.py#L172
It would be nice if you could use multiple TaskHandlers at the same time: e.g. to use [S3TaskHandler](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/log/s3_task_handler/index.html#airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler) and a custom logging TaskHandler, so that logs are written to both.
My particular use case is that I want all logs to be written to S3, and that a subset of the logs is also written to Kafka. Currently this is not possible without forking Airflow.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18604: Allow multiple simultaneous remote logging methods
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18604:
URL: https://github.com/apache/airflow/issues/18604#issuecomment-930278642
Yeah. I thin this is not a new feature - it is possible with advanced logging and I do not think we need to add a feature for it. Maybe documentation update with explicit mentioning that you can enable multiple loggers this way might be useful. I am thinking about adding such documentation after several discussions I had recently.
For now I will convert it into discussion as I do not think there is a feature coming out of that (we can always move back to an issue if we get to conclusion that we should).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #18604: Allow multiple simultaneous remote logging methods
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #18604:
URL: https://github.com/apache/airflow/issues/18604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #18604: Allow multiple simultaneous remote logging methods
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #18604:
URL: https://github.com/apache/airflow/issues/18604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18604: Allow multiple simultaneous remote logging methods
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18604:
URL: https://github.com/apache/airflow/issues/18604#issuecomment-930278642
Yeah. I thin this is not a new feature - it is possible with advanced logging and I do not think we need to add a feature for it. Maybe documentation update with explicit mentioning that you can enable multiple loggers this way might be useful. I am thinking about adding such documentation after several discussions I had recently.
For now I will convert it into discussion as I do not think there is a feature coming out of that (we can always move back to an issue if we get to conclusion that we should).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #18604: Allow multiple simultaneous remote logging methods
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #18604:
URL: https://github.com/apache/airflow/issues/18604#issuecomment-930065851
I don't think this is quite a generic case for us to support it, but you are free to make any modifications to the logger configuration. Here is docs about it:
http://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #18604: Allow multiple simultaneous remote logging methods
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #18604:
URL: https://github.com/apache/airflow/issues/18604#issuecomment-930065851
I don't think this is quite a generic case for us to support it, but you are free to make any modifications to the logger configuration. Here is docs about it:
http://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org