[jira] (IMPALA-15447) Iceberg scan telemetry (Stream A + B) to report Iceberg scan metrics.

["Prabhat Mishra (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-15447 ]


    Prabhat Mishra deleted comment on IMPALA-15447:
    -----------------------------------------

was (Author: JIRAUSER314792):
https://docs.google.com/document/d/1esQWpVWJs13tb5VgvHGU-KSn9tX73rhm8av9K0sPzwI/edit?usp=sharing

> Iceberg scan telemetry (Stream A + B) to report Iceberg scan metrics.
> ---------------------------------------------------------------------
>
>                 Key: IMPALA-15447
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15447
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Catalog, Frontend
>            Reporter: Prabhat Mishra
>            Assignee: Prabhat Mishra
>            Priority: Major
>
> Adds two-stream Iceberg query telemetry to the Impala frontend (Java/fe):
> Stream A — Scan Reports (IcebergScanPlanner.java, IcebergUtil.java):
>  - Wires a catalog-configured side MetricsReporter alongside the existing InMemoryMetricsReporter in IcebergScanPlanner. The existing InMemoryMetricsReporter is kept because it feeds addIcebergScanMetricsToProfile() (query profile).
>  - The side reporter is loaded once per JVM (double-checked locking, volatile) via CatalogUtil.loadMetricsReporter, driven by the metrics-reporter-impl JVM system property
>  - For the fast-path where planFiles() is skipped, synthesizes a faithful ScanReport from already-computed file descriptors and posts it only to the side reporter (InMemory reporter and query profile are left unchanged).
>  - Stamps a query-id and engine-name onto every scan report via a new IcebergUtil.planFiles overload that passes scan options as report metadata, enabling cross-stream correlation downstream.
> Stream B — Query Shape (IcebergQueryShapeCollector.java):
>  - New class IcebergQueryShapeCollector runs once per query, right after Planner.createPlans.
>  - Walks the plan tree, extracts equi-join keys and ORDER BY columns that resolve to Iceberg tables.
>  - POSTs a per-table query-shape event via the same telemetry library (called reflectively to avoid a compile-time dependency), tagged with the identical query-id used in Stream A.
>  - Hooked into Frontend.createExecRequest (Frontend.java).
> Both streams are entirely best-effort: neither throws nor blocks query planning, and both no-op when no side reporter is configured (preserving today's default behavior byte-for-byte).
> —



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org