[jira] [Created] (IMPALA-11171) Impala still re-reads Iceberg manifest files for each SCAN node.

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

Zoltán Borók-Nagy created IMPALA-11171:
------------------------------------------

             Summary: Impala still re-reads Iceberg manifest files for each SCAN node.
                 Key: IMPALA-11171
                 URL: https://issues.apache.org/jira/browse/IMPALA-11171
             Project: IMPALA
          Issue Type: Bug
          Components: Frontend
            Reporter: Zoltán Borók-Nagy


In IcebergUtil.getIcebergDataFiles() we issue scan.planFiles():
https://github.com/apache/impala/blob/7f1ce039be30d5b36a490e8b07728f82f5d4c3de/fe/src/main/java/org/apache/impala/util/IcebergUtil.java#L534

scan.planFiles() needs to read the manifest files to return a list of files to be scanned. This unfortunately adds significant overhead to the plan time for short-running queries.

Maybe we can do the followings to mitigate this issue:
* cache TableScan.planFiles() without predicates being used, and use this instead of pushing predicates to Iceberg. It would need a logic to decide when to use the cached plan files and when to push down predicates
* Figure out if it is possible to cache manifest files so we don't need to re-read them for each table scan.
** If this is not possible then we might need to contribute code to Iceberg




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org