You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/08/13 18:04:35 UTC
[PR] Document how to separate metrics from replicated Airflow components [airflow]
1fanwang opened a new pull request, #71564:
URL: https://github.com/apache/airflow/pull/71564
Running schedulers in high availability is the recommended production setup, but every Airflow process reports the same `service.name`. Nothing distinguishes one scheduler's telemetry from another's, so all replicas publish to the same series and the backend keeps whichever export arrived last.
It shows up most clearly on gauges. Each scheduler samples the metadata database on its own loop, so `pool.open_slots` and its siblings settle on an arbitrary replica's sample and look like they are flapping. Operators reasonably read that as a broken metric.
OpenTelemetry already solves this with `service.instance.id`, and Airflow already honours it — the meter provider is built with `Resource.create()`, which merges `OTEL_RESOURCE_ATTRIBUTES`. No code change is needed. But the metrics docs never mention high availability at all, so there is nothing to lead an operator to it.
This adds a short section to the OpenTelemetry docs covering why replicas collide, the environment variable that separates them, aggregating across replicas at query time, the `instance` label mapping for Prometheus-compatible backends, and the fact that StatsD has no equivalent.
# Testing Done
The documented mechanism was run rather than assumed.
**1. Airflow really does merge the variable.** This is the claim the whole section rests on:
```console
$ OTEL_RESOURCE_ATTRIBUTES="service.instance.id=scheduler-a" python -c "
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
r = Resource.create(attributes={SERVICE_NAME: 'airflow'})
for k, v in sorted(r.attributes.items()): print(f'{k} = {v}')"
service.instance.id = scheduler-a
service.name = airflow
telemetry.sdk.language = python
telemetry.sdk.name = opentelemetry
telemetry.sdk.version = 1.44.0
```
`Resource.create()` there is the same call the meter provider makes in `otel_logger.py`.
**2. It actually separates the replicas.** Two `SafeOtelLogger` instances built the way `get_otel_logger()` builds one, differing only in `service.instance.id`, each reporting a different `open_slots` for the same pool:
<details><summary>Script and output</summary>
```python
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from airflow_shared.observability.metrics.otel_logger import SafeOtelLogger
def build_scheduler(instance_id):
resource = Resource.create(
attributes={SERVICE_NAME: "airflow", "service.instance.id": instance_id}
)
reader = InMemoryMetricReader()
return SafeOtelLogger(otel_provider=MeterProvider(metric_readers=[reader], resource=resource)), reader
tags = {"pool_name": "default_pool"}
schedulers = [build_scheduler("scheduler-a"), build_scheduler("scheduler-b")]
for (scheduler, _), open_slots in zip(schedulers, (128, 126)):
scheduler.gauge("pool.open_slots", open_slots, tags=tags)
for _, reader in schedulers:
rm = reader.get_metrics_data().resource_metrics[0]
instance = rm.resource.attributes["service.instance.id"]
for metric in rm.scope_metrics[0].metrics:
point = next(iter(metric.data.data_points))
print(f"{metric.name}{{service.instance.id={instance}, {tags}}} = {point.value}")
```
```
airflow.pool.open_slots{service.instance.id=scheduler-a, {'pool_name': 'default_pool'}} = 128
airflow.pool.open_slots{service.instance.id=scheduler-b, {'pool_name': 'default_pool'}} = 126
```
Both samples survive as separate series, so `min by (pool_name)` returns 126 rather than an arbitrary one of the two. Without the attribute the second export overwrites the first.
</details>
**3. The page itself.** Both linked specs return `200`. `prek run --files airflow-core/docs/.../metrics.rst` passes, including the RST and codespell hooks. Parsing the file with docutils reports no structural problems — the only messages are the `envvar`, `doc` and `ref` roles at lines 45, 54 and 294, which are Sphinx roles that predate this change. I could not run the full `breeze build-docs` render locally, since it wanted an interactive CI image rebuild; CI covers it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3798852288
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
Review Comment:
outdated removed in a rewrite https://github.com/apache/airflow/pull/71564/commits/f6214bbc380e01ccdb4923971090715c80215aaa based on https://github.com/apache/airflow/pull/71564#discussion_r3796351150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3806144478
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +104,51 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+.. _identifying-components-and-their-instances:
+
+Identifying components and their instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+OpenTelemetry labels each metric with the resource that produced it. Two resource attributes
+decide how much of a deployment can be told apart:
+
+``service.name``
+ Which component reported the metric. It defaults to ``airflow`` for every Airflow process, so
+ a scheduler, a triggerer and a worker arrive under one name. Set it per component to attribute
+ a metric to the kind of process that produced it.
+
+``service.instance.id``
+ Which process of that component reported the metric. It is unset by default, so processes
+ running the same component are indistinguishable. Set it per process to attribute a metric to
+ one of them.
+
+Airflow reads ``service.name`` from ``OTEL_SERVICE_NAME``, and every other resource attribute from
+``OTEL_RESOURCE_ATTRIBUTES``:
+
+.. code-block:: bash
+
+ # on one of the schedulers
+ export OTEL_SERVICE_NAME="airflow-scheduler"
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Processes that share a resource also share a series, and the backend keeps whichever export
+arrived last. Where several processes run the same component this loses data rather than
Review Comment:
The suggested wording does look a LOT better then before, it was indeed confusing.
Done in https://github.com/apache/airflow/pull/71564/commits/b32b9e4d55be9162c2e9445238c439116a7a7ee3 and https://github.com/apache/airflow/pull/71564/commits/3a2e3dda8a269e3238c397c8c2cfb588099a3443
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3799956948
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
Review Comment:
agreed this is confusing, fixed in https://github.com/apache/airflow/pull/71564/changes/4ef1a18d8ba9003c77954541de7f774fb89ee417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3798854145
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3806135092
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +104,51 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+.. _identifying-components-and-their-instances:
+
+Identifying components and their instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+OpenTelemetry labels each metric with the resource that produced it. Two resource attributes
+decide how much of a deployment can be told apart:
+
+``service.name``
+ Which component reported the metric. It defaults to ``airflow`` for every Airflow process, so
+ a scheduler, a triggerer and a worker arrive under one name. Set it per component to attribute
+ a metric to the kind of process that produced it.
+
+``service.instance.id``
+ Which process of that component reported the metric. It is unset by default, so processes
+ running the same component are indistinguishable. Set it per process to attribute a metric to
Review Comment:
Done in https://github.com/apache/airflow/pull/71564/commits/b32b9e4d55be9162c2e9445238c439116a7a7ee3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3799960397
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
Review Comment:
fixed in https://github.com/apache/airflow/pull/71564/changes/4ef1a18d8ba9003c77954541de7f774fb89ee417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on PR #71564:
URL: https://github.com/apache/airflow/pull/71564#issuecomment-5347737049
@1fanwang, it is in the history of the PR description, but the latest edit to it made it not follow our guidelines. Could you fix that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3804766046
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -53,6 +53,12 @@ custom StatsD client. This module must be available on your :envvar:`PYTHONPATH`
See :doc:`../modules_management` for details on how Python and Airflow manage modules.
+.. note::
+
+ StatsD has no resource concept, so metrics cannot be attributed to the process that
+ produced them. Use OpenTelemetry if you need that — see
Review Comment:
Should we also say that in scheduler HA it's recommended to use OTel otherwise, you end up with the values from the scheduler that exports last?
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +104,51 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+.. _identifying-components-and-their-instances:
+
+Identifying components and their instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+OpenTelemetry labels each metric with the resource that produced it. Two resource attributes
+decide how much of a deployment can be told apart:
+
+``service.name``
+ Which component reported the metric. It defaults to ``airflow`` for every Airflow process, so
+ a scheduler, a triggerer and a worker arrive under one name. Set it per component to attribute
+ a metric to the kind of process that produced it.
+
+``service.instance.id``
+ Which process of that component reported the metric. It is unset by default, so processes
+ running the same component are indistinguishable. Set it per process to attribute a metric to
Review Comment:
```suggestion
running the same component (e.g. 2+ schedulers) are indistinguishable. Set it per process to attribute a metric to
```
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +104,51 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+.. _identifying-components-and-their-instances:
+
+Identifying components and their instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+OpenTelemetry labels each metric with the resource that produced it. Two resource attributes
+decide how much of a deployment can be told apart:
+
+``service.name``
+ Which component reported the metric. It defaults to ``airflow`` for every Airflow process, so
+ a scheduler, a triggerer and a worker arrive under one name. Set it per component to attribute
+ a metric to the kind of process that produced it.
+
+``service.instance.id``
+ Which process of that component reported the metric. It is unset by default, so processes
+ running the same component are indistinguishable. Set it per process to attribute a metric to
+ one of them.
+
+Airflow reads ``service.name`` from ``OTEL_SERVICE_NAME``, and every other resource attribute from
+``OTEL_RESOURCE_ATTRIBUTES``:
+
+.. code-block:: bash
+
+ # on one of the schedulers
+ export OTEL_SERVICE_NAME="airflow-scheduler"
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Processes that share a resource also share a series, and the backend keeps whichever export
+arrived last. Where several processes run the same component this loses data rather than
Review Comment:
> Where several processes run the same component this loses data rather
This is confusing. Maybe you meant `when`.
```suggestion
arrived last. When several processes run the same component, data are lost instead of aggregated.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #71564:
URL: https://github.com/apache/airflow/pull/71564#issuecomment-5347732208
Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to **[Airflow 3.3.2](https://github.com/apache/airflow/milestone/150)** based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.
> This comment was generated by [Milestone Tag Assistant](https://github.com/apache/airflow/blob/main/.github/workflows/milestone-tag-assistant.yml).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how to separate metrics from replicated Airflow components [airflow]
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3796251773
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
Review Comment:
`service.name` and `service.instance.id` aren't the same. So for example, if I have this in my docker-compose file
```yaml
airflow-scheduler:
<<: *airflow-common
command: scheduler
environment:
<<: *airflow-common-env
OTEL_SERVICE_NAME: airflow-scheduler
OTEL_RESOURCE_ATTRIBUTES: service.instance.id=airflow-scheduler-1
...
airflow-scheduler-2:
<<: *airflow-common
command: scheduler
environment:
<<: *airflow-common-env
OTEL_SERVICE_NAME: airflow-scheduler
OTEL_RESOURCE_ATTRIBUTES: service.instance.id=airflow-scheduler-2
...
```
Then in prometheus I would get
```
airflow_scheduler_critical_section_duration{
exported_instance="airflow-scheduler-1", exported_job="airflow-scheduler",
instance="otel-collector:8889", job="otel-collector",
otel_scope_name="airflow.metrics.otel_logger"
}
```
and
```
airflow_scheduler_critical_section_duration{
exported_instance="airflow-scheduler-2", exported_job="airflow-scheduler",
instance="otel-collector:8889", job="otel-collector",
otel_scope_name="airflow.metrics.otel_logger"
}
```
I think it's worth distinguishing and mentioning both.
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
Review Comment:
This is confusing. There isn't actual replication because we don't have a leader-follower relationship between the schedulers. They are both independent.
It doesn't necessarily have to do with scheduler HA, because there is value in distinguishing between different components such as worker and scheduler as well.
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
Review Comment:
> Gauges are where this shows up in the data.
We should be more generic and describe what will happen if they don't use distinct service ids.
We should say that the OpenTelemetry service name defaults to `Airflow` if unset and explain when it's good to set it to something else. Also explain why and when to set the `service.instance.id`.
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
Review Comment:
You use the word `replica` and then refer to a scheduler, triggerer, dag processor, which is inaccurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3798858140
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
+
+.. code-block:: bash
+
+ # on each scheduler, triggerer or Dag processor
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Airflow merges ``OTEL_RESOURCE_ATTRIBUTES`` into the OpenTelemetry resource, so nothing else needs
+configuring. Replicas can then be aggregated at query time instead of overwriting one another —
+for example, the lowest number of open slots any scheduler observed:
+
+.. code-block:: text
+
+ min by (pool_name) (airflow_pool_open_slots)
+
+Backends implementing the OpenTelemetry `Prometheus compatibility
+<https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/>`_ spec expose
+``service.instance.id`` as the ``instance`` label.
Review Comment:
Reframed in https://github.com/apache/airflow/commit/f6214bbc380e01ccdb4923971090715c80215aaa. It now leads with the fact that how the attributes surface depends on the backend, and cites the OpenTelemetry Prometheus-compatibility spec rather than any specific backend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl merged PR #71564:
URL: https://github.com/apache/airflow/pull/71564
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3798866294
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
Review Comment:
Thanks! agreed,
it's `service.name` from `OTEL_SERVICE_NAME` (falling back to the `otel_service` option, then `airflow`), `service.instance.id` from `OTEL_RESOURCE_ATTRIBUTES`
done in https://github.com/apache/airflow/commit/f6214bbc380e01ccdb4923971090715c80215aaa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #71564:
URL: https://github.com/apache/airflow/pull/71564#issuecomment-5336194576
+1, LGTM!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on PR #71564:
URL: https://github.com/apache/airflow/pull/71564#issuecomment-5366147713
> @1fanwang, it is in the history of the PR description, but the latest edit to it made it not follow our guidelines. Could you fix that?
Thanks! Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #71564:
URL: https://github.com/apache/airflow/pull/71564#issuecomment-5347737939
### Backport successfully created: v3-3-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/pull/71854"><img src="https://img.shields.io/badge/PR-71854-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how to separate metrics from replicated Airflow components [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on PR #71564:
URL: https://github.com/apache/airflow/pull/71564#issuecomment-5305006845
> Could you update your PR description to follow our project guidelines - https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst?
Thanks for the pointer, updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3798838321
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
+
+.. code-block:: bash
+
+ # on each scheduler, triggerer or Dag processor
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Airflow merges ``OTEL_RESOURCE_ATTRIBUTES`` into the OpenTelemetry resource, so nothing else needs
+configuring. Replicas can then be aggregated at query time instead of overwriting one another —
+for example, the lowest number of open slots any scheduler observed:
+
+.. code-block:: text
+
+ min by (pool_name) (airflow_pool_open_slots)
+
+Backends implementing the OpenTelemetry `Prometheus compatibility
+<https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/>`_ spec expose
+``service.instance.id`` as the ``instance`` label.
+
+.. note::
+
+ This applies to OpenTelemetry only. The StatsD protocol has no resource concept, so metrics
+ from replicated components cannot be distinguished this way.
Review Comment:
[Done in f6214bb](https://github.com/apache/airflow/pull/71564/commits/f6214bbc380e01ccdb4923971090715c80215aaa)
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
+
+.. code-block:: bash
+
+ # on each scheduler, triggerer or Dag processor
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Airflow merges ``OTEL_RESOURCE_ATTRIBUTES`` into the OpenTelemetry resource, so nothing else needs
+configuring. Replicas can then be aggregated at query time instead of overwriting one another —
+for example, the lowest number of open slots any scheduler observed:
+
+.. code-block:: text
+
+ min by (pool_name) (airflow_pool_open_slots)
+
+Backends implementing the OpenTelemetry `Prometheus compatibility
+<https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/>`_ spec expose
+``service.instance.id`` as the ``instance`` label.
+
+.. note::
+
+ This applies to OpenTelemetry only. The StatsD protocol has no resource concept, so metrics
+ from replicated components cannot be distinguished this way.
Review Comment:
Done in https://github.com/apache/airflow/pull/71564/commits/f6214bbc380e01ccdb4923971090715c80215aaa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3798838321
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
+
+.. code-block:: bash
+
+ # on each scheduler, triggerer or Dag processor
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Airflow merges ``OTEL_RESOURCE_ATTRIBUTES`` into the OpenTelemetry resource, so nothing else needs
+configuring. Replicas can then be aggregated at query time instead of overwriting one another —
+for example, the lowest number of open slots any scheduler observed:
+
+.. code-block:: text
+
+ min by (pool_name) (airflow_pool_open_slots)
+
+Backends implementing the OpenTelemetry `Prometheus compatibility
+<https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/>`_ spec expose
+``service.instance.id`` as the ``instance`` label.
+
+.. note::
+
+ This applies to OpenTelemetry only. The StatsD protocol has no resource concept, so metrics
+ from replicated components cannot be distinguished this way.
Review Comment:
Done in f6214bb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3798843972
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
+
+.. code-block:: bash
+
+ # on each scheduler, triggerer or Dag processor
Review Comment:
done in https://github.com/apache/airflow/pull/71564/commits/f6214bbc380e01ccdb4923971090715c80215aaa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on PR #71564:
URL: https://github.com/apache/airflow/pull/71564#issuecomment-5320354776
> @1fanwang Thanks for the PR. This is a valid issue that could occur with multiple schedulers and this PR fills the documentation gap.
>
> We should explain the differences between `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES: service.instance.id=` and when to use each and why.
>
> The text should be more generic rather than focus on the discussed issue.
>
> When the users read the section, they should get the idea that when configuring OTel, they might also need to set a distinct service name and id but they should also understand when and why.
Good point, rewritten along those lines in https://github.com/apache/airflow/commit/f6214bbc380e01ccdb4923971090715c80215aaa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3799963086
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
Review Comment:
Restructured as a definition list so each attribute maps to what it identifies https://github.com/apache/airflow/pull/71564/changes/4ef1a18d8ba9003c77954541de7f774fb89ee417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document how to separate metrics from replicated Airflow components [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3791397577
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
+
+.. code-block:: bash
+
+ # on each scheduler, triggerer or Dag processor
Review Comment:
```suggestion
# On every Airflow component which has more than one instance, like scheduler, triggerer, etc.
```
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
Review Comment:
```suggestion
replica of a component — Schedulers in high availability, several Triggerers or Dag Processors —
```
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
+
+.. code-block:: bash
+
+ # on each scheduler, triggerer or Dag processor
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Airflow merges ``OTEL_RESOURCE_ATTRIBUTES`` into the OpenTelemetry resource, so nothing else needs
+configuring. Replicas can then be aggregated at query time instead of overwriting one another —
+for example, the lowest number of open slots any scheduler observed:
+
+.. code-block:: text
+
+ min by (pool_name) (airflow_pool_open_slots)
+
+Backends implementing the OpenTelemetry `Prometheus compatibility
+<https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/>`_ spec expose
+``service.instance.id`` as the ``instance`` label.
Review Comment:
Not sure if we should have it here - it is not the only compatibility backend with OpenTelemetry, and I'm not sure if we should promote one over another by mentioning it in our docs.
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their own series:
+
+.. code-block:: bash
+
+ # on each scheduler, triggerer or Dag processor
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Airflow merges ``OTEL_RESOURCE_ATTRIBUTES`` into the OpenTelemetry resource, so nothing else needs
+configuring. Replicas can then be aggregated at query time instead of overwriting one another —
+for example, the lowest number of open slots any scheduler observed:
+
+.. code-block:: text
+
+ min by (pool_name) (airflow_pool_open_slots)
+
+Backends implementing the OpenTelemetry `Prometheus compatibility
+<https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/>`_ spec expose
+``service.instance.id`` as the ``instance`` label.
+
+.. note::
+
+ This applies to OpenTelemetry only. The StatsD protocol has no resource concept, so metrics
+ from replicated components cannot be distinguished this way.
Review Comment:
So we could just move this section under `OpenTelemetry` and remove that note, or add a note to `StatsD` that if there is a need for separating information regarding pods, the user has to use OpenTelemetry.
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g. ``airflow.cfg``
`SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment running more than one
+replica of a component — schedulers in high availability, several triggerers or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and its siblings keep
Review Comment:
```suggestion
own loop and exports a value for the same series, so e.g. ``pool.open_slots`` and its siblings keep
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document service.name and service.instance.id for OpenTelemetry metrics [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3806131401
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -53,6 +53,12 @@ custom StatsD client. This module must be available on your :envvar:`PYTHONPATH`
See :doc:`../modules_management` for details on how Python and Airflow manage modules.
+.. note::
+
+ StatsD has no resource concept, so metrics cannot be attributed to the process that
+ produced them. Use OpenTelemetry if you need that — see
Review Comment:
Makes sense! Clarified in https://github.com/apache/airflow/pull/71564/commits/b32b9e4d55be9162c2e9445238c439116a7a7ee3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org