You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "moomindani (via GitHub)" <gi...@apache.org> on 2026/04/30 06:50:32 UTC
[I] Add OpenTelemetry MetricsReporter for vendor-neutral metric export [iceberg]
moomindani opened a new issue, #16169:
URL: https://github.com/apache/iceberg/issues/16169
### Motivation
`iceberg-core` currently ships only two `MetricsReporter` implementations:
- `LoggingMetricsReporter` — verbose `toString` to a log file
- `RESTMetricsReporter` — REST catalog only
Users on Hive Metastore, Glue, JDBC, or Hadoop catalogs effectively have no
out-of-the-box way to ship `ScanReport` / `CommitReport` to an observability
backend. Several existing issues and PRs touch this gap:
- #14490, #14875, #7278 — requests / discussions around metrics export
- #16107 — a GCP-specific `CloudMonitoringMetricsReporter` that uses
reflection to access internal types
A standard OTLP-based `MetricsReporter` would give users one well-known
integration point and let any OTel Collector fan metrics out from there
(Prometheus, CloudWatch, Datadog, Grafana Cloud, Honeycomb, etc.) without
maintaining vendor-specific code in Iceberg itself.
### Proposed design
A new `org.apache.iceberg.metrics.OtelMetricsReporter` implementing
`MetricsReporter` with the standard no-arg constructor +
`initialize(Map<String,String>)` pattern.
**Catalog properties**
| Property | Default | Notes |
|---|---|---|
| `otel.endpoint` | `http://localhost:4318/v1/metrics` (HTTP), `http://localhost:4317` (gRPC) | OTLP endpoint URL |
| `otel.protocol` | `http/protobuf` | `http/protobuf` or `grpc` |
| `otel.service-name` | `iceberg` | OTel resource attribute |
| `otel.export-interval-ms` | `60000` | `PeriodicMetricReader` interval |
| `otel.token` | — | Bearer auth shorthand: adds `Authorization: Bearer <token>` |
| `otel.headers` | — | Comma-separated `key=value` pairs; supports JWT padding (`=` in value) |
**Metric mapping**
Each `ScanReport` and `CommitReport` field maps to a stable metric name with
attributes that match existing Iceberg conventions:
- `iceberg.scan.planning.duration` (histogram, ms)
- `iceberg.scan.result.{data_files,delete_files}` (sum)
- `iceberg.scan.data_manifests.{scanned,skipped}` (sum)
- `iceberg.scan.file_size.bytes` (sum, By)
- `iceberg.commit.duration` (histogram, ms)
- `iceberg.commit.{attempts,records.added}` (sum)
- `iceberg.commit.data_files.{added,removed}` (sum)
- `iceberg.commit.file_size.added_bytes` (sum, By)
Attributes: `iceberg.table.name`, `iceberg.snapshot.id`, `iceberg.schema.id`,
`iceberg.operation`.
**Dependencies**
OpenTelemetry SDK 1.61.0 added as `compileOnly` so users who do not configure
the reporter pay no runtime classpath cost. Test scope adds
`opentelemetry-sdk-testing` for `InMemoryMetricReader`-based unit tests.
JDK 11+, no breaking changes.
### Validation
I have a working prototype on a local branch and have validated it E2E
against two completely different OTLP backends, using the **same reporter
class without modification** — to my mind the most important property of the
design:
1. **Databricks Zerobus Ingest** (OTLP/gRPC, Bearer auth) — metrics land
directly in a Unity Catalog Delta table; verified with SQL aggregations
matching injected values exactly.
2. **Amazon CloudWatch** (OTLP/HTTP, SigV4 via OTel Collector) — same
reporter, same metric names, same attributes; verified via PromQL
`sum by()` and ratio queries.
Detailed validation reports (commands, queries, gotchas) can be shared on
request or attached to the PR.
### Disclosure
I used Claude Code to help draft and prototype this work. I reviewed every
change by hand and ran the full test/lint loop locally before each
iteration; the validation results above are from my own runs against real
backends. Per the project's
[AI-assisted contribution guidelines](https://iceberg.apache.org/contribute/#guidelines-for-ai-assisted-contributions),
I am opening this Issue first to gather design feedback before opening a PR.
### Open questions
1. Is the preferred home `iceberg-core` (with `compileOnly` deps as drafted),
or a separate `iceberg-opentelemetry` module to keep `iceberg-core` lean?
2. Naming — does `otel.*` match expectations? Should `otel.token` exist as a
first-class property, or should users always use `otel.headers`?
3. Should the metric naming convention follow the
[OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/)
for database / lakehouse semconv (none defined yet for table-format ops),
or stay on the `iceberg.*` namespace as sketched above?
4. Anything else I should resolve before opening a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: Add OpenTelemetry MetricsReporter [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on issue #16169:
URL: https://github.com/apache/iceberg/issues/16169#issuecomment-6618895267
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: Add OpenTelemetry MetricsReporter [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on issue #16169:
URL: https://github.com/apache/iceberg/issues/16169#issuecomment-4357408410
Supporting OpenTelemetry seems reasonable. I opened #14360 last year to add OTEL support to the REST catalog.
cc: @singhpk234 @jbonofre
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: Add OpenTelemetry MetricsReporter [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on issue #16169:
URL: https://github.com/apache/iceberg/issues/16169#issuecomment-4353956575
Hi @moomindani I will also give Otel support to Core. Thank you for this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org