You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "wangsheng (Code Review)" <ge...@cloudera.org> on 2020/11/27 09:04:36 UTC

[Impala-ASF-CR] IMPALA-10361: Supported using filed id to resolve columns for Iceberg tables

wangsheng has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16788


Change subject: IMPALA-10361: Supported using filed id to resolve columns for Iceberg tables
......................................................................

IMPALA-10361: Supported using filed id to resolve columns for Iceberg tables

We supported resolve column by field id for Iceberg table in this
patch. We can use 'set PARQUET_FALLBACK_SCHEMA_RESOLUTION=FIELDID'
or 'set PARQUET_FALLBACK_SCHEMA_RESOLUTION=2' to choose field id
resolving. But pay attention, if you use this for non-Iceberg
table, the result will be NULL.

Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
---
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/types.cc
M be/src/runtime/types.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/Types.thrift
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
18 files changed, 182 insertions(+), 11 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16788/1
-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 1
Gerrit-Owner: wangsheng <sk...@163.com>

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.

Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................

Patch Set 5: Code-Review+1

(2 comments)

Thanks for adding the tests, the change looks great. I'm planning to do another round next week, so only giving it +1 for now.

I think for Iceberg tables we should always try to resolve columns via field id, i.e. for Iceberg tables we can ignore the value of PARQUET_FALLBACK_SCHEMA_RESOLUTION.

Do you plan to implement this for ORC tables as well (in a separate patch)? Maybe we should open another Jira/subtask for that.

http://gerrit.cloudera.org:8080/#/c/16788/5/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java:

http://gerrit.cloudera.org:8080/#/c/16788/5/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java@339
PS5, Line 339: boolean isFullAcidTable = AcidUtils.isFullAcidTable(msTbl.getParameters());
Iceberg tables cannot be full ACID, maybe it can be a precondition.

http://gerrit.cloudera.org:8080/#/c/16788/5/testdata/data/README
File testdata/data/README:

http://gerrit.cloudera.org:8080/#/c/16788/5/testdata/data/README@608
PS5, Line 608: generated file will contains multi blocks, multi pages per block.
Please add information about the newly added files and tests.

--
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 5
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 04 Dec 2020 13:19:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.

Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................

IMPALA-10361: Use field id to resolve columns for Iceberg tables

We supported resolve column by field id for Iceberg table in this
patch. Currently, we use field id to resolve column for Iceberg
tables, which means 'PARQUET_FALLBACK_SCHEMA_RESOLUTION' is invalid
for Iceberg tables.

Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Reviewed-on: http://gerrit.cloudera.org:8080/16788
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
---
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch-serialize-test.cc
M be/src/runtime/types.cc
M be/src/runtime/types.h
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/Types.thrift
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/StructType.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/data/README
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00001-1-13d79bd6-4b97-4680-b4e1-52e93b6ce04e-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00006-6-305c9b7a-f42d-4245-b806-dfa7a792593f-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00009-9-224fe2d6-b0d9-42d6-bc95-15f52ecb29ad-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00017-17-1a38e294-5992-48d9-a18e-08e129bb418c-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00023-23-74cfcf22-3de2-489a-b1ec-d5141e75a8e8-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00027-27-5f91dc85-b8f3-4cc2-a5c6-38b7fee49709-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00030-30-dc3510cc-e765-43bc-be03-c5561a8d50a3-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00031-31-364afc4a-b718-406d-a532-58fab5c8f85d-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00004-4-7a1a8e89-8aeb-4405-be64-76557432cf21-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00014-14-765d552a-fddc-42f3-adfd-ecba20a01d80-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00015-15-9957db43-3b9a-4a50-9946-d003cc1d461c-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00019-19-1e1895d0-1f42-4c30-989f-968802831077-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00020-20-bb59ac6d-aeee-4c35-9f8a-1a03127d33b8-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00028-28-44ba3ad9-737c-4416-a32c-501cc9a4aa90-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00003-3-31478795-ff6a-4a20-9fff-8dc4907c1ba7-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00007-7-3b4a35cb-22b5-4a5d-932b-89f222b0b2c7-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00011-11-a40c4c8c-de16-487a-89b3-ee8e58b4fd07-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00012-12-ed4ba706-2ace-4d41-b475-6bda3ba72306-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00022-22-57f5e226-5f8f-49b9-b998-039b8362b7a0-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00025-25-5fec3538-052d-493e-9479-b59fc8aece0f-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/4ceee3ab-8653-423c-b8ac-0ad5f7b0579b-m0.avro
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/snap-8063525763252762178-1-4ceee3ab-8653-423c-b8ac-0ad5f7b0579b.avro
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v1.metadata.json
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v2.metadata.json
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-resolution-by-name.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/query_test/test_scanners.py
55 files changed, 724 insertions(+), 22 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Zoltan Borok-Nagy: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6756/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 11
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 10 Dec 2020 11:17:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.

wangsheng has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................

IMPALA-10361: Use field id to resolve columns for Iceberg tables

We supported resolve column by field id for Iceberg table in this
patch. Currently, we use field id to resolve column for Iceberg
tables, which means 'PARQUET_FALLBACK_SCHEMA_RESOLUTION' is invalid
for Iceberg tables.

Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
---
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch-serialize-test.cc
M be/src/runtime/types.cc
M be/src/runtime/types.h
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/Types.thrift
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/StructType.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/data/README
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00001-1-13d79bd6-4b97-4680-b4e1-52e93b6ce04e-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00006-6-305c9b7a-f42d-4245-b806-dfa7a792593f-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00009-9-224fe2d6-b0d9-42d6-bc95-15f52ecb29ad-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00017-17-1a38e294-5992-48d9-a18e-08e129bb418c-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00023-23-74cfcf22-3de2-489a-b1ec-d5141e75a8e8-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00027-27-5f91dc85-b8f3-4cc2-a5c6-38b7fee49709-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00030-30-dc3510cc-e765-43bc-be03-c5561a8d50a3-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00031-31-364afc4a-b718-406d-a532-58fab5c8f85d-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00004-4-7a1a8e89-8aeb-4405-be64-76557432cf21-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00014-14-765d552a-fddc-42f3-adfd-ecba20a01d80-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00015-15-9957db43-3b9a-4a50-9946-d003cc1d461c-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00019-19-1e1895d0-1f42-4c30-989f-968802831077-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00020-20-bb59ac6d-aeee-4c35-9f8a-1a03127d33b8-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00028-28-44ba3ad9-737c-4416-a32c-501cc9a4aa90-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00003-3-31478795-ff6a-4a20-9fff-8dc4907c1ba7-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00007-7-3b4a35cb-22b5-4a5d-932b-89f222b0b2c7-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00011-11-a40c4c8c-de16-487a-89b3-ee8e58b4fd07-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00012-12-ed4ba706-2ace-4d41-b475-6bda3ba72306-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00022-22-57f5e226-5f8f-49b9-b998-039b8362b7a0-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00025-25-5fec3538-052d-493e-9479-b59fc8aece0f-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/4ceee3ab-8653-423c-b8ac-0ad5f7b0579b-m0.avro
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/snap-8063525763252762178-1-4ceee3ab-8653-423c-b8ac-0ad5f7b0579b.avro
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v1.metadata.json
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v2.metadata.json
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-resolution-by-name.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/query_test/test_scanners.py
54 files changed, 723 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16788/11
-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 11
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7773/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 4
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 04 Dec 2020 03:40:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................


Patch Set 7: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6734/


-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 7
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 08 Dec 2020 07:35:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10361: Supported using field id to resolve columns for Iceberg tables

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Supported using field id to resolve columns for Iceberg tables
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7741/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 1
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Nov 2020 09:26:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.

wangsheng has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................

IMPALA-10361: Use field id to resolve columns for Iceberg tables

We supported resolve column by field id for Iceberg table in this
patch. Currently, we use field id to resolve column for Iceberg
tables, which means 'PARQUET_FALLBACK_SCHEMA_RESOLUTION' is invalid
for Iceberg tables.

Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
---
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch-serialize-test.cc
M be/src/runtime/types.cc
M be/src/runtime/types.h
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/Types.thrift
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/StructType.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/data/README
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00001-1-13d79bd6-4b97-4680-b4e1-52e93b6ce04e-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00006-6-305c9b7a-f42d-4245-b806-dfa7a792593f-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00009-9-224fe2d6-b0d9-42d6-bc95-15f52ecb29ad-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00017-17-1a38e294-5992-48d9-a18e-08e129bb418c-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00023-23-74cfcf22-3de2-489a-b1ec-d5141e75a8e8-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00027-27-5f91dc85-b8f3-4cc2-a5c6-38b7fee49709-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00030-30-dc3510cc-e765-43bc-be03-c5561a8d50a3-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00031-31-364afc4a-b718-406d-a532-58fab5c8f85d-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00004-4-7a1a8e89-8aeb-4405-be64-76557432cf21-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00014-14-765d552a-fddc-42f3-adfd-ecba20a01d80-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00015-15-9957db43-3b9a-4a50-9946-d003cc1d461c-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00019-19-1e1895d0-1f42-4c30-989f-968802831077-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00020-20-bb59ac6d-aeee-4c35-9f8a-1a03127d33b8-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00028-28-44ba3ad9-737c-4416-a32c-501cc9a4aa90-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00003-3-31478795-ff6a-4a20-9fff-8dc4907c1ba7-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00007-7-3b4a35cb-22b5-4a5d-932b-89f222b0b2c7-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00011-11-a40c4c8c-de16-487a-89b3-ee8e58b4fd07-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00012-12-ed4ba706-2ace-4d41-b475-6bda3ba72306-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00022-22-57f5e226-5f8f-49b9-b998-039b8362b7a0-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00025-25-5fec3538-052d-493e-9479-b59fc8aece0f-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/4ceee3ab-8653-423c-b8ac-0ad5f7b0579b-m0.avro
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/snap-8063525763252762178-1-4ceee3ab-8653-423c-b8ac-0ad5f7b0579b.avro
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v1.metadata.json
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v2.metadata.json
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-resolution-by-name.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/query_test/test_scanners.py
55 files changed, 724 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16788/12
-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 12
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>

[Impala-ASF-CR] IMPALA-10361: Supported using field id to resolve columns for Iceberg tables

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.

Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Supported using field id to resolve columns for Iceberg tables
......................................................................


Patch Set 2:

(5 comments)

Thanks WangSheng for working on this important task! I did a first pass and the change looks good to me. Looking forward for the tests.

http://gerrit.cloudera.org:8080/#/c/16788/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/16788/2//COMMIT_MSG@7
PS2, Line 7: Supported
nit: Use field id to resolve columns for Iceberg tables


http://gerrit.cloudera.org:8080/#/c/16788/2//COMMIT_MSG@11
PS2, Line 11: to choose field id
            : resolving.
I think this should be the default for Iceberg tables.


http://gerrit.cloudera.org:8080/#/c/16788/2/common/thrift/ImpalaInternalService.thrift
File common/thrift/ImpalaInternalService.thrift:

http://gerrit.cloudera.org:8080/#/c/16788/2/common/thrift/ImpalaInternalService.thrift@48
PS2, Line 48: FIELDID
nit: FIELD_ID


http://gerrit.cloudera.org:8080/#/c/16788/2/fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java
File fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java:

http://gerrit.cloudera.org:8080/#/c/16788/2/fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java@25
PS2, Line 25: 
nit: add comment


http://gerrit.cloudera.org:8080/#/c/16788/2/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:

http://gerrit.cloudera.org:8080/#/c/16788/2/fe/src/main/java/org/apache/impala/catalog/Table.java@439
PS2, Line 439: By
nit: From



-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 2
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 30 Nov 2020 12:38:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.

Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................


Patch Set 7: Code-Review+1

Thanks for the modifications, WangSheng! The change LGTM once the BE test issues is resolved.


-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 7
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 08 Dec 2020 17:27:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................


Patch Set 12: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 12
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 10 Dec 2020 18:01:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10361: Supported using field id to resolve columns for Iceberg tables

Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.

wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Supported using field id to resolve columns for Iceberg tables
......................................................................


Patch Set 2:

Hi Zoltan,Gabor. I've already implement a first version for resolving columns by field id. I created a simple test table in my environment for array/struct/map, it worked in normal catalog mode, but return null for struct type in local catalog mode, I'm still working on this patch. Hope you can give me some suggestions, Thanks a lot.


-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 2
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 27 Nov 2020 09:31:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.

wangsheng has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................

IMPALA-10361: Use field id to resolve columns for Iceberg tables

We supported resolve column by field id for Iceberg table in this
patch. We can use 'set PARQUET_FALLBACK_SCHEMA_RESOLUTION=FIELD_ID'
or 'set PARQUET_FALLBACK_SCHEMA_RESOLUTION=2' to choose field id
resolving. But pay attention, if you use this for non-Iceberg
table, the result will be NULL.

Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
---
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/types.cc
M be/src/runtime/types.h
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/Types.thrift
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/StructType.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00001-1-13d79bd6-4b97-4680-b4e1-52e93b6ce04e-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00006-6-305c9b7a-f42d-4245-b806-dfa7a792593f-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00009-9-224fe2d6-b0d9-42d6-bc95-15f52ecb29ad-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00017-17-1a38e294-5992-48d9-a18e-08e129bb418c-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00023-23-74cfcf22-3de2-489a-b1ec-d5141e75a8e8-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00027-27-5f91dc85-b8f3-4cc2-a5c6-38b7fee49709-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00030-30-dc3510cc-e765-43bc-be03-c5561a8d50a3-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-08/action=view/00031-31-364afc4a-b718-406d-a532-58fab5c8f85d-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00004-4-7a1a8e89-8aeb-4405-be64-76557432cf21-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00014-14-765d552a-fddc-42f3-adfd-ecba20a01d80-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00015-15-9957db43-3b9a-4a50-9946-d003cc1d461c-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00019-19-1e1895d0-1f42-4c30-989f-968802831077-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00020-20-bb59ac6d-aeee-4c35-9f8a-1a03127d33b8-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-09/action=click/00028-28-44ba3ad9-737c-4416-a32c-501cc9a4aa90-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00003-3-31478795-ff6a-4a20-9fff-8dc4907c1ba7-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00007-7-3b4a35cb-22b5-4a5d-932b-89f222b0b2c7-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00011-11-a40c4c8c-de16-487a-89b3-ee8e58b4fd07-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00012-12-ed4ba706-2ace-4d41-b475-6bda3ba72306-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00022-22-57f5e226-5f8f-49b9-b998-039b8362b7a0-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/data/event_time_hour=2020-01-01-10/action=download/00025-25-5fec3538-052d-493e-9479-b59fc8aece0f-00000.parquet
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/4ceee3ab-8653-423c-b8ac-0ad5f7b0579b-m0.avro
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/snap-8063525763252762178-1-4ceee3ab-8653-423c-b8ac-0ad5f7b0579b.avro
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v1.metadata.json
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/v2.metadata.json
A testdata/data/iceberg_test/hadoop_catalog/iceberg_resolution_test/functional_parquet/iceberg_resolution_test/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/query_test/test_scanners.py
51 files changed, 701 insertions(+), 20 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16788/4
-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 4
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>

[Impala-ASF-CR] IMPALA-10361: Use field id to resolve columns for Iceberg tables

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Use field id to resolve columns for Iceberg tables
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7824/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 11
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 10 Dec 2020 11:35:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10361: Supported using field id to resolve columns for Iceberg tables

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 )

Change subject: IMPALA-10361: Supported using field id to resolve columns for Iceberg tables
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7742/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435
Gerrit-Change-Number: 16788
Gerrit-PatchSet: 2
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 27 Nov 2020 09:43:07 +0000
Gerrit-HasComments: No