[jira] [Commented] (IMPALA-11171) Impala still re-reads Iceberg manifest files for each SCAN node.

["Riza Suminto (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17505369#comment-17505369 ] 

Riza Suminto commented on IMPALA-11171:
---------------------------------------

Are we suppose to cache BaseTable and reuse it for each impala scan node planning?
I see method called BaseSnapshot.cacheManifest() in one of my stacktrace capture.

> Impala still re-reads Iceberg manifest files for each SCAN node.
> ----------------------------------------------------------------
>
>                 Key: IMPALA-11171
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11171
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Frontend
>            Reporter: Zoltán Borók-Nagy
>            Priority: Major
>              Labels: impala-iceberg
>         Attachments: Screen Shot 2022-03-12 at 3.23.28 PM.png
>
>
> In IcebergUtil.getIcebergDataFiles() we issue scan.planFiles():
> https://github.com/apache/impala/blob/7f1ce039be30d5b36a490e8b07728f82f5d4c3de/fe/src/main/java/org/apache/impala/util/IcebergUtil.java#L534
> scan.planFiles() needs to read the manifest files to return a list of files to be scanned. This unfortunately adds significant overhead to the plan time for short-running queries.
> Maybe we can do the followings to mitigate this issue:
> * cache TableScan.planFiles() without predicates being used, and use this instead of pushing predicates to Iceberg. It would need a logic to decide when to use the cached plan files and when to push down predicates
> * Figure out if it is possible to cache manifest files so we don't need to re-read them for each table scan.
> ** If this is not possible then we might need to contribute code to Iceberg



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org