You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/01/18 16:58:00 UTC
[jira] [Commented] (IMPALA-11826) Avoid calling planFiles() on Iceberg V2 tables when there are no predicates
[ https://issues.apache.org/jira/browse/IMPALA-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678326#comment-17678326 ]
ASF subversion and git services commented on IMPALA-11826:
----------------------------------------------------------
Commit 1e1b8f25b686471b088148dd296a2eb731160302 in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1e1b8f25b ]
IMPALA-11826: Avoid calling planFiles() on Iceberg V2 tables when there are no predicates
Similar to IMPALA-11591 but this Jira extends it to V2 tables. With
this patch we group data files into two categories in
IcebergContentFileStore:
* data files without deletes
* data files with deletes
With this information we can avoid calling planFiles() when planning
the scans of Iceberg tables. We can just set the lists of the file
descriptors based on IcebergContentFileStore then invoke the regular
planning methods.
iceberg-v2-tables.test had to be updated a bit because now we are
calculating the lengths of the file paths based on Impala's file
descriptor objects + table location, and not based on data file
information in Iceberg metadata (which has the file system prefix
stripped)
Testing:
* executed existing tests
* Updated plan tests
Change-Id: Ia46bd2dce248a9e096fc1c0bd914fc3fa4686fb0
Reviewed-on: http://gerrit.cloudera.org:8080/19419
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Avoid calling planFiles() on Iceberg V2 tables when there are no predicates
> ---------------------------------------------------------------------------
>
> Key: IMPALA-11826
> URL: https://issues.apache.org/jira/browse/IMPALA-11826
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Similar to IMPALA-11591 but this Jira extends it to V2 tables.
> We can extend IcebergContentFileStore to group data files:
> * data files without corresponding delete files
> * data files with delete files
> Once we have this we don't have to invoke planFiles() on V2 table SCANs without predicates. We can just simply create the usual UNION - ANTI JOIN plan based on IcebergContentFileStore:
> {noformat}
> UNION ALL
> / \
> / \
> SCAN ANTI JOIN
> data files / \
> without / \
> deletes SCAN SCAN
> data files delete files
> with deletes{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org