You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "howardyoo (via GitHub)" <gi...@apache.org> on 2024/02/27 18:13:29 UTC
[I] [AIP-49] Airflow support for OTEL traces [airflow]
howardyoo opened a new issue, #37752:
URL: https://github.com/apache/airflow/issues/37752
### Description
We have had a successful release of OTEL support for Airflow that covered `metrics` part. Now, the time is right to move into the next chapter, where we'd be adding the new Tracing support for Airflow. Tracing will complement the additional details that was not present in metrics. Metrics are lightweight and does have its usefulness in detecting the overall health status of Airflow. However, when you try to dig deeper into debugging the root cause, metrics often does not contain that detail and you have to drill down with things like logs and traces.
OTEL traces also cover the interactions between different airflow components together and shows the wholistic picture of how each DAG run flew through schedulers, dag processors, workers, and eventually executors, providing the necessary linkages to understand how a single cycle of DAG run ran, and what kind of components were involved, and ultimately show how each task results ended up with.
The last part of the OTEL implementation would involve `logging` support which will have logging handlers that will emit log messages using OTEL standard to an OTEL compatible endpoint such as otel collector. However, this is yet another separate initiative that is not yet part of the scope of this work, and should follow after the tracing support is done.
for more details related to this, please look for [AIP-49](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow).
### Use case/motivation
For each dag runs initiated and ran, Airflow should keep track of its progress in full cycle (from start of the DAG to finish), and emit traces and spans pertaining to the OpenTelemetry specification, using OTEL SDK and API. The configuration should also have options to configure how the tracing would work.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [AIP-49] Airflow support for OTEL traces [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #37752: [AIP-49] Airflow support for OTEL traces
URL: https://github.com/apache/airflow/issues/37752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [AIP-49] Airflow support for OTEL traces [airflow]
Posted by "KimboTodd (via GitHub)" <gi...@apache.org>.
KimboTodd commented on issue #37752:
URL: https://github.com/apache/airflow/issues/37752#issuecomment-2089067948
We need better monitoring in astronomer and this would help a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [AIP-49] Airflow support for OTEL traces [airflow]
Posted by "howardyoo (via GitHub)" <gi...@apache.org>.
howardyoo commented on issue #37752:
URL: https://github.com/apache/airflow/issues/37752#issuecomment-2125955919
Yes, of cource, @woody1872 !
I have a PR made ready for this, and am awaiting any abled airflow contributors to review and approve. Still waiting, though.
https://github.com/apache/airflow/pull/37948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [AIP-49] Airflow support for OTEL traces [airflow]
Posted by "woody1872 (via GitHub)" <gi...@apache.org>.
woody1872 commented on issue #37752:
URL: https://github.com/apache/airflow/issues/37752#issuecomment-2125803474
Hi @howardyoo
Just wanted to check if this work still planned?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org