[Impala-ASF-CR] IMPALA-13267: Display number of partitions for Iceberg tables (WIP)

["Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>]

Noemi Pap-Takacs has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/23113 )

Change subject: IMPALA-13267: Display number of partitions for Iceberg tables (WIP)
......................................................................

IMPALA-13267: Display number of partitions for Iceberg tables (WIP)

Before this change, query plans and profile reported only a single
partition even for partitioned Iceberg tables, which was misleading
for users.
Now we can display the number of scanned partitions correctly for
both partitioned and unpartitioned Iceberg tables. This is achieved by
extracting the partition values from the file descriptors and storing
them in the IcebergContentFileStore. Instead of storing this information
redundantly in all file descriptors, we store them in one place and
reference the partition metadata in the FDs with an id.
This also gives the opportunity to optimize memory consumption in the
Catalog and Coordinator as well as reduce network traffic between them.

Time travel is handled similarly to oldFileDescMap. In that case the
output is [Num scanned partitions]/[Num partitions in current snapshot].

Testing:
 - Planner tests
 - E2E tests
   - partition transforms
   - partition evolution
   - DROP PARTITION
   - time travel

Change-Id: Ifb2f654bc6c9bdf9cfafc27b38b5ca2f7b6b4872
---
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-merge-insert-only.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-merge.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-mixed-file-format.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-optimize.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-predicates-disabled-subsetting.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-predicates.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-delete.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables-hash-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables-resources.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables.test
M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-update.test
M testdata/workloads/functional-planner/queries/PlannerTest/tablesample-iceberg.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitions.test
M tests/query_test/test_iceberg.py
27 files changed, 643 insertions(+), 351 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/23113/5
-- 
To view, visit http://gerrit.cloudera.org:8080/23113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifb2f654bc6c9bdf9cfafc27b38b5ca2f7b6b4872
Gerrit-Change-Number: 23113
Gerrit-PatchSet: 5
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>