You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Zoltán Borók-Nagy (Jira)" <ji...@apache.org> on 2022/11/04 12:11:00 UTC
[jira] [Resolved] (IMPALA-11171) Impala still re-reads Iceberg manifest files for each SCAN node.
[ https://issues.apache.org/jira/browse/IMPALA-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zoltán Borók-Nagy resolved IMPALA-11171.
----------------------------------------
Fix Version/s: Impala 4.2.0
Resolution: Fixed
> Impala still re-reads Iceberg manifest files for each SCAN node.
> ----------------------------------------------------------------
>
> Key: IMPALA-11171
> URL: https://issues.apache.org/jira/browse/IMPALA-11171
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 4.2.0
>
> Attachments: Screen Shot 2022-03-12 at 3.23.28 PM.png
>
>
> In IcebergUtil.getIcebergDataFiles() we issue scan.planFiles():
> https://github.com/apache/impala/blob/7f1ce039be30d5b36a490e8b07728f82f5d4c3de/fe/src/main/java/org/apache/impala/util/IcebergUtil.java#L534
> scan.planFiles() needs to read the manifest files to return a list of files to be scanned. This unfortunately adds significant overhead to the plan time for short-running queries.
> Maybe we can do the followings to mitigate this issue:
> * cache TableScan.planFiles() without predicates being used, and use this instead of pushing predicates to Iceberg. It would need a logic to decide when to use the cached plan files and when to push down predicates
> * Figure out if it is possible to cache manifest files so we don't need to re-read them for each table scan.
> ** If this is not possible then we might need to contribute code to Iceberg
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org