[D] How to define custom metrics [airflow]

["aeroyorch (via GitHub)" <gi...@apache.org>]

GitHub user aeroyorch created a discussion: How to define custom metrics

Is there any recommended way to define and emit custom metrics in Airflow?

For example using custom stats like:

```python
from airflow.sdk.observability import stats

stats.gauge("my_service.queue_depth", 42)
# ...and similarly stats.incr(...) / stats.decr(...) / stats.timing(...) / stats.timer(...)
```

If this is the case, a dedicated "Custom Metrics" section in `airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst` would be very helpful.

GitHub link: https://github.com/apache/airflow/discussions/69096

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How to define custom metrics [airflow]

["aeroyorch (via GitHub)" <gi...@apache.org>]

GitHub user aeroyorch added a comment to the discussion: How to define custom metrics

@apoorva-01 if you can take a look

GitHub link: https://github.com/apache/airflow/discussions/69096#discussioncomment-17531057

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How to define custom metrics [airflow]

["apoorva-01 (via GitHub)" <gi...@apache.org>]

GitHub user apoorva-01 added a comment to the discussion: How to define custom metrics

Yeah, that's basically the supported way. In Airflow 3 the metrics client moved to the task SDK, and the import you wrote is the right one:

```python
from airflow.sdk.observability import stats

stats.gauge("my_service.queue_depth", 42)
stats.incr("my_service.processed")
stats.decr("my_service.in_flight")
stats.timing("my_service.batch_ms", 1234)      # ms or a timedelta
with stats.timer("my_service.batch"):          # times the block
    ...
```

`from airflow.stats import Stats` still works but it's deprecated now (it just warns and forwards to the same place), so the `airflow.sdk.observability` path is the one to use going forward.

The one gotcha that trips people up: these calls are no-ops unless you actually have a metrics backend turned on. Out of the box nothing is emitting, so your gauge goes nowhere and it looks broken. Turn one on in config:

```ini
[metrics]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
```

or the OpenTelemetry equivalent with `otel_on = True`.

Also watch `[metrics] metrics_allow_list` / `metrics_block_list` if you've set them. Allow-list is prefix-matched, so if it's configured and your `my_service.*` prefix isn't in it, the custom metric gets filtered out silently (and if you set an allow-list, the block-list is ignored entirely). Signatures if you need the rest: `incr/decr(stat, count=None, rate=None, *, tags=...)`, `gauge(stat, value, *, tags=...)`, all take a `tags` dict for dimensional metrics.

Agree a dedicated "Custom Metrics" section in metrics.rst would help, this comes up a lot.

GitHub link: https://github.com/apache/airflow/discussions/69096#discussioncomment-17523512

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How to define custom metrics [airflow]

["aeroyorch (via GitHub)" <gi...@apache.org>]

GitHub user aeroyorch edited a comment on the discussion: How to define custom metrics

@apoorva-01 if you can take a look to https://github.com/apache/airflow/pull/69371

GitHub link: https://github.com/apache/airflow/discussions/69096#discussioncomment-17531057

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How to define custom metrics [airflow]

["aeroyorch (via GitHub)" <gi...@apache.org>]

GitHub user aeroyorch added a comment to the discussion: How to define custom metrics

Thanks! I mainly wanted to confirm that my understanding was correct. I don't have any problem adding these updates to the metrics docs :).

GitHub link: https://github.com/apache/airflow/discussions/69096#discussioncomment-17523636

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How to define custom metrics [airflow]

["aeroyorch (via GitHub)" <gi...@apache.org>]

GitHub user aeroyorch edited a comment on the discussion: How to define custom metrics

@apoorva-01 if you can take a look: https://github.com/apache/airflow/pull/69371

GitHub link: https://github.com/apache/airflow/discussions/69096#discussioncomment-17531057

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How to define custom metrics [airflow]

["aeroyorch (via GitHub)" <gi...@apache.org>]

GitHub user aeroyorch closed a discussion: How to define custom metrics

Is there any recommended way to define and emit custom metrics in Airflow?

For example using custom stats like:

```python
from airflow.sdk.observability import stats

stats.gauge("my_service.queue_depth", 42)
# ...and similarly stats.incr(...) / stats.decr(...) / stats.timing(...) / stats.timer(...)
```

If this is the case, a dedicated "Custom Metrics" section in `airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst` would be very helpful.

GitHub link: https://github.com/apache/airflow/discussions/69096

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org