You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/05/12 16:39:00 UTC
[PR] feat(metrics/otel): add histogram bucket views per metric-name pattern [airflow]
1fanwang opened a new pull request, #66805:
URL: https://github.com/apache/airflow/pull/66805
### Problem
Follow-up to #64207, which set `ExponentialBucketHistogramAggregation` as the
default for every OTel `Histogram` instrument. That covers timer/timing.
Non-timer histograms (`*_count`, `*_duration` on direct `Stats.histogram()`,
custom `*_delay` metrics) still pick bucket boundaries at each call site,
so the same family of metrics can end up with different bucket shapes
depending on which module created the instrument.
### Fix
New `shared/observability/.../metrics/histogram_buckets.py` module that:
- Defines three named bucket aggregations — `LATENCY_BUCKETS` (exponential),
`COUNT_BUCKETS` (small linear), `DELAY_BUCKETS` (large-range).
- Exposes `DEFAULT_PATTERN_BUCKETS` mapping `*_duration` / `*_delay` /
`*_count` to those aggregations.
- Provides `build_views_for_patterns()` returning OTel `View` objects keyed
by instrument-name glob. Deployments can pass an override dict.
`otel_logger.get_otel_logger()` layers the pattern views on top of the
existing instrument-type baseline view — the per-instrument-type
exponential default still applies, the pattern views only refine it for
matching names.
### Tests
`test_histogram_buckets.py` covers the default pattern map, the per-pattern
aggregation resolution, custom-mapping override, and that
`build_views_for_patterns()` returns one `View` per entry with the right
`instrument_name`.
`test_otel_logger.py::test_get_otel_logger_uses_exponential_histogram_view`
updated to assert the layered shape (baseline view + pattern views).
### Design note
Flagging this as a design discussion. A reasonable counter-proposal is
"let each call site pass its bucket boundaries" rather than centralising
the map. Happy to take that direction if maintainers prefer.
Closes #66801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics: add histogram bucket views per metric-name pattern [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang closed pull request #66805: Metrics: add histogram bucket views per metric-name pattern
URL: https://github.com/apache/airflow/pull/66805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics: add histogram bucket views per metric-name pattern [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #66805:
URL: https://github.com/apache/airflow/pull/66805#issuecomment-5173218163
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics: add histogram bucket views per metric-name pattern [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #66805:
URL: https://github.com/apache/airflow/pull/66805#discussion_r3772746169
##########
shared/observability/src/airflow_shared/observability/metrics/otel_logger.py:
##########
@@ -507,16 +512,21 @@ def get_otel_logger(
except (ImportError, AttributeError):
pass
+ # Order matters: OTel applies views in sequence, so the name-pattern views
+ # must follow the instrument-type baseline to override it for matching names.
+ histogram_views: list[View] = [
+ View(
+ instrument_type=metrics.Histogram,
+ aggregation=ExponentialBucketHistogramAggregation(),
+ ),
+ *build_views_for_patterns(),
+ ]
Review Comment:
Good catch, just checked otel SDK 1.44.0: `_handle_view_instrument_match` already appends a match per matching view, no override needed. Closing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics: add histogram bucket views per metric-name pattern [airflow]
Posted by "aaron-y-chen (via GitHub)" <gi...@apache.org>.
aaron-y-chen commented on code in PR #66805:
URL: https://github.com/apache/airflow/pull/66805#discussion_r3733064168
##########
shared/observability/src/airflow_shared/observability/metrics/otel_logger.py:
##########
@@ -507,16 +512,21 @@ def get_otel_logger(
except (ImportError, AttributeError):
pass
+ # Order matters: OTel applies views in sequence, so the name-pattern views
+ # must follow the instrument-type baseline to override it for matching names.
+ histogram_views: list[View] = [
+ View(
+ instrument_type=metrics.Histogram,
+ aggregation=ExponentialBucketHistogramAggregation(),
+ ),
+ *build_views_for_patterns(),
+ ]
Review Comment:
It seems OTel applies every matching view independently, there's no later-overrides-earlier.
A `*_count` instrument matches both the baseline and its pattern view, so it silently exports two same-named streams, one exponential and one explicit. I don't think it's working as expected, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org