You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Owen-CH-Leung (via GitHub)" <gi...@apache.org> on 2024/12/17 01:58:21 UTC
[PR] Add write feature to ESTaskHandler [airflow]
Owen-CH-Leung opened a new pull request, #44973:
URL: https://github.com/apache/airflow/pull/44973
TBC
related: #42780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add write feature to ESTaskHandler [airflow]
Posted by "Owen-CH-Leung (via GitHub)" <gi...@apache.org>.
Owen-CH-Leung commented on PR #44973:
URL: https://github.com/apache/airflow/pull/44973#issuecomment-2565732017
@potiuk may I also seek your review for my PR ? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add write feature to ESTaskHandler [airflow]
Posted by "Owen-CH-Leung (via GitHub)" <gi...@apache.org>.
Owen-CH-Leung commented on code in PR #44973:
URL: https://github.com/apache/airflow/pull/44973#discussion_r1888457249
##########
docs/apache-airflow-providers-elasticsearch/logging/index.rst:
##########
@@ -22,6 +22,8 @@ Writing logs to Elasticsearch
Airflow can be configured to read task logs from Elasticsearch and optionally write logs to stdout in standard or json format. These logs can later be collected and forwarded to the Elasticsearch cluster using tools like fluentd, logstash or others.
+Airflow also supports writing log to Elasticsearch directly without requiring additional software like filebeat and logstash. To enable this feature, set ``write_to_es`` and ``json_format`` to ``True`` and ``write_stdout`` to ``False`` in ``airflow.cfg``.
Review Comment:
Indeed, while in this PR I have added feature to automatically write task logs to ES, I've also added codes to consider the `delete_local_logs` defined in airflow config.
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#delete-local-logs
If users set both `write_to_es` and `delete_local_logs` to true, airflow will attempt to remove local task logs upon successfully writing task logs to ES. This is inspired by the `S3TaskHandler` which shares the same feature.
https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py#L100-L102
I've revised the doc to flag this out.
Actually I'm not sure if we should consider the `delete_local_logs` config to remove local task logs. Maybe it's safer to not remove local copies so in case writing to es fails, users can always re-enable file-beat to ship logs. But Incorporating the delete_local_logs config aligns this feature more closely with the behavior of other handlers and provides a more comprehensive solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add write feature to ESTaskHandler [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #44973:
URL: https://github.com/apache/airflow/pull/44973#discussion_r1887963370
##########
docs/apache-airflow-providers-elasticsearch/logging/index.rst:
##########
@@ -22,6 +22,8 @@ Writing logs to Elasticsearch
Airflow can be configured to read task logs from Elasticsearch and optionally write logs to stdout in standard or json format. These logs can later be collected and forwarded to the Elasticsearch cluster using tools like fluentd, logstash or others.
+Airflow also supports writing log to Elasticsearch directly without requiring additional software like filebeat and logstash. To enable this feature, set ``write_to_es`` and ``json_format`` to ``True`` and ``write_stdout`` to ``False`` in ``airflow.cfg``.
Review Comment:
Is there a downside for such setup that we need to mention? For example: sine no filebeat and logstash. Is there a risk of losing some records?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add write feature to ESTaskHandler [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #44973:
URL: https://github.com/apache/airflow/pull/44973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org