You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Nándor Kollár (Jira)" <ji...@apache.org> on 2026/10/05 12:37:00 UTC
[jira] [Commented] (IMPALA-11577) Optimize getting stored file types for Iceberg tables
[ https://issues.apache.org/jira/browse/IMPALA-11577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123348#comment-18123348 ]
Nándor Kollár commented on IMPALA-11577:
----------------------------------------
I'm not sure that this worth the effort. The size of the [change|https://gerrit.cloudera.org/c/24736/] is moderate, however we win just milliseconds if we skip iterating through the file list:
Results Summary
{code:java}
┌────────────┬───────────────────┬───────────────────┬────────────┐
│ File Count │ Old (iterate all) │ New (precomputed) │ Speedup │
├────────────┼───────────────────┼───────────────────┼────────────┤
│ 100 │ 717 ns │ 27 ns │ 26x │
├────────────┼───────────────────┼───────────────────┼────────────┤
│ 1,000 │ 9.6 us │ 27 ns │ 354x │
├────────────┼───────────────────┼───────────────────┼────────────┤
│ 10,000 │ 126 us │ 31 ns │ 4,055x │
├────────────┼───────────────────┼───────────────────┼────────────┤
│ 100,000 │ 2.8 ms │ 30 ns │ 95,480x │
├────────────┼───────────────────┼───────────────────┼────────────┤
│ 1,000,000 │ 25.2 ms │ 28 ns │ 899,815x │
├────────────┼───────────────────┼───────────────────┼────────────┤
│ 10,000,000 │ 197 ms │ 30 ns │ 6,654,766x │
└────────────┴───────────────────┴───────────────────┴────────────┘
{code}
Although the speedup looks significant, we're talking about milliseconds here, which we pay once, during planning time.
> Optimize getting stored file types for Iceberg tables
> -----------------------------------------------------
>
> Key: IMPALA-11577
> URL: https://issues.apache.org/jira/browse/IMPALA-11577
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Gergely Fürnstáhl
> Assignee: Nándor Kollár
> Priority: Minor
> Labels: impala-iceberg
>
> Spawned from IMPALA-10610
> Impala supports mixed file formats for Iceberg tables, which means every file can have different file format and it uses the set of existing file formats for planning purposes. Currently Impala goes through all file's metadata to aggregate this information, which can be slow if there are lots of data files.
> We could optimized this by storing this aggregated information somewhere (e.g. in Iceberg - yet to be implemented - [https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/SnapshotSummary.java])
> Update:
> IcebergContentFileStore supports aggregating file formats, similarly as the proposed change in iceberg, but might not be the correct approach. It represents the state of the current snapshot, but IcebergScanNode receives a possibly time travelled/pruned version of the file descriptor list.
> Further optimization ideas:
> * Use the aggregated info from the iceberg SnapshotSummary/IcebergContentFileStore if possible (e.g. current snapshot, no pruning)
> * Exit the loop if we found all the available file formats (might be to costly if we check the condition it in every iteration)
> * Parallelize the aggregation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org