[jira] [Updated] (HIVE-29781) Iceberg: Basic stats fetch optimization - bulk partition stats read with query-scoped cache

["Denys Kuzmenko (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-29781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Denys Kuzmenko updated HIVE-29781:
----------------------------------
    Fix Version/s: 4.3.0
       Resolution: Fixed
           Status: Resolved  (was: Patch Available)

> Iceberg: Basic stats fetch optimization - bulk partition stats read with query-scoped cache
> -------------------------------------------------------------------------------------------
>
>                 Key: HIVE-29781
>                 URL: https://issues.apache.org/jira/browse/HIVE-29781
>             Project: Hive
>          Issue Type: Bug
>          Components: Iceberg integration
>    Affects Versions: 4.2.0
>            Reporter: Denys Kuzmenko
>            Assignee: Denys Kuzmenko
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
>  1. Per-partition stats fetch is O(file) per partition.
> With hive.iceberg.stats.source=iceberg, StatsUtils.collectStatistics builds BasicStats per pruned partition. Each one called getPartishSummary, which opens the entire partition-stats file from object storage and linearly scans it for that single partition. N partitions ⇒ N full file opens + O(N²/2) record parses per TableScan.                                                                                                                                                                   
>   2. The stats worker pool can't use the table-metadata cache (HIVE-24313 interaction). The per-partition work runs on the dedicated basic-stats-* ForkJoin pool, whose thread factory doesn't propagate the  SessionState ThreadLocal. Without it, SessionStateUtil.getQueryState(conf) is empty, so the handler's getTable() takes skipCache = orElse(true) — a full Catalogs.loadTable per partition (HMS RPC + metadata.json read from storage). That's each "Refreshing table metadata" line. The session-level cache is both unreadable and unpopulatable from those threads, so it never warms. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)