[GitHub] [airflow] malthe commented on issue #16706: Improve operational logging and telemetry reporting in hooks

[GitBox <gi...@apache.org>]

malthe commented on issue #16706:
URL: https://github.com/apache/airflow/issues/16706#issuecomment-908309524


   I think perhaps a starting point is to contribute "opentelemetry-instrumentation-X" packages (e.g. [opentelemetry-instrumentation-psycopg2](https://pypi.org/project/opentelemetry-instrumentation-psycopg2/)).
   
   It should be possible then to [add a span around task execution](https://opentelemetry.io/docs/java/manual_instrumentation/#create-nested-spans) to logically group an arbitrary and unknown set of events and/or metrics produced by would-be instrumented libraries used during the execution of a single task.
   
   The next step might then be to add a metrics/events tab to the task instance page and display events/metrics emitted during the execution – for example, if exporting telemetry to Prometheus, then we can use the [HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/#expression-queries) to pull this data dynamically.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org