You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Prabhat Mishra (Jira)" <ji...@apache.org> on 2026/09/28 16:48:00 UTC

[jira] [Created] (IMPALA-15447) Iceberg scan telemetry (Stream A + B) to report Iceberg scan metrics.

Prabhat Mishra created IMPALA-15447:
---------------------------------------

             Summary: Iceberg scan telemetry (Stream A + B) to report Iceberg scan metrics.
                 Key: IMPALA-15447
                 URL: https://issues.apache.org/jira/browse/IMPALA-15447
             Project: IMPALA
          Issue Type: Improvement
          Components: Catalog, Frontend
            Reporter: Prabhat Mishra


Description:

Adds two-stream Iceberg query telemetry to the Impala frontend (Java/fe):

Stream A — Scan Reports (IcebergScanPlanner.java, IcebergUtil.java):
- Wires a catalog-configured side MetricsReporter alongside the existing InMemoryMetricsReporter in IcebergScanPlanner. The existing InMemoryMetricsReporter is kept because it feeds addIcebergScanMetricsToProfile() (query profile).
- The side reporter is loaded once per JVM (double-checked locking, volatile) via CatalogUtil.loadMetricsReporter, driven by the metrics-reporter-impl JVM system property set through Cloudera Manager.
- For the fast-path where planFiles() is skipped, synthesizes a faithful ScanReport from already-computed file descriptors and posts it only to the side reporter (InMemory reporter and query profile are left unchanged).
- Stamps a query-id and engine-name onto every scan report via a new IcebergUtil.planFiles overload that passes scan options as report metadata, enabling cross-stream correlation downstream.

Stream B — Query Shape (IcebergQueryShapeCollector.java):
- New class IcebergQueryShapeCollector runs once per query, right after Planner.createPlans.
- Walks the plan tree, extracts equi-join keys and ORDER BY columns that resolve to Iceberg tables.
- POSTs a per-table query-shape event via the same telemetry library (called reflectively to avoid a compile-time dependency), tagged with the identical query-id used in Stream A.
- Hooked into Frontend.createExecRequest (Frontend.java).

Both streams are entirely best-effort: neither throws nor blocks query planning, and both no-op when no side reporter is configured (preserving today's default behavior byte-for-byte).

---



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org