You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "wangsheng (Code Review)" <ge...@cloudera.org> on 2020/07/06 07:20:10 UTC
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
wangsheng has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16143
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
IMPALA-9741: Supported query icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently support
PARQUET and ORC. And if you don't identity this property in your SQL,
default file format is PARQUT.
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
23 files changed, 842 insertions(+), 135 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/1
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 1
Gerrit-Owner: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
IMPALA-9741: Supported query Icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't identity this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When query iceberg table, we pushdown
partition column predicates to iceberg to decided which data files
need to be scanned, and then transformed these information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/custom_cluster/test_iceberg.py
M tests/query_test/test_scanners.py
83 files changed, 1,320 insertions(+), 161 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/8
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 8
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 31: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 31
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sat, 05 Sep 2020 20:54:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 25:
> That's great news! It looks like some of the Iceberg-related tests
> failed - https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11883/
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11884/
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3067/
>
> But the good news is that it loaded the data successfully.
Yes Tim, I regenerated the test files, but forgot to modify the test cases in iceberg-query.test which contains 'show files in xxx' related to specific files‘ name. I will adjust code and submit Jenkins to verify as soon as possible.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 06:32:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 27:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7063/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 27
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 14:57:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6354/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 09:16:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 29:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7104/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 04 Sep 2020 12:34:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 25:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6366/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 11:15:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6615/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 10
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 16 Jul 2020 06:58:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#22). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
45 files changed, 1,436 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/22
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 22
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 25:
That's great news! It looks like some of the Iceberg-related tests failed - https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11883/ https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11884/ https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3067/
But the good news is that it loaded the data successfully.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 04:09:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 18:
(10 comments)
One thing I didn't understand about the patch is the support for partition pruning. I don't see enough code here to handle all the edge cases - partition evolution, different partition transforms, etc.
I'm OK with deferring optimizations to a later patch, but want to make sure that this patch is correct and that we document what the limitations are.
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@611
PS18, Line 611: // Iceberg table only has one partition spec now
What does this mean? We don't support partition evolution yet?
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@615
PS18, Line 615: filedName
nit: fieldName
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1889
PS18, Line 1889: public
Can you make this protected instead of public, since it's only being used within this package.
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@372
PS18, Line 372: public
I think this can be protected
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@61
PS18, Line 61: icebergConjunts_
nit: icebergConjuncts_
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@94
PS18, Line 94: Icber
nit: Iceberg
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@123
PS18, Line 123: it.remove();
I think this predicate pushdown needs a bit more explanation.
As far as I can see there are three possibilities for predicate pushdown:
1. The column is not part of any Iceberg partition expression so cannot be pushed down at all.
2. The column is part of all partition keys without any transformation (i.e. the identity transformation). https://iceberg.apache.org/spec/#partitioning - the predicate can be pushed down and does not need to be removed at all.
3. The column is part of some or all partition keys, or is transformed in some way during partitioning - we can push down the predicate to get partition pruning, but also need to evaluate it in the scan. I guess the two cases here are that there has been some partition evolution over time, and that there is some transform of the partition column.
It's not clear how the code is handling #3. I think this needs to be clarified.
The strict vs inclusive projection in the spec might be useful for explaining it - https://iceberg.apache.org/spec/#scan-planning
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@226
PS18, Line 226: * Returns true if this column does not belong to iceberg partition column.
What does this mean in terms of the iceberg spec - https://iceberg.apache.org/spec/#partitioning. Does this mean that it only returns true if the column is part of the partition with an identity transform?
What about if partition evolution has happened and it is a partition column for some but not all columns?
http://gerrit.cloudera.org:8080/#/c/16143/18/testdata/data/README
File testdata/data/README:
http://gerrit.cloudera.org:8080/#/c/16143/18/testdata/data/README@508
PS18, Line 508: <<<<<<< HEAD
nit: remove this merge marker
http://gerrit.cloudera.org:8080/#/c/16143/18/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test:
http://gerrit.cloudera.org:8080/#/c/16143/18/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test@3
PS18, Line 3: Icebreg
nit: Iceberg
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 18
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 10 Aug 2020 17:53:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
IMPALA-9741: Support quering Icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When query iceberg table, we pushdown
partition column predicates to iceberg to decided which data files
need to be scanned, and then transformed these information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/custom_cluster/test_iceberg.py
M tests/query_test/test_scanners.py
83 files changed, 1,320 insertions(+), 161 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/9
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 9
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 26:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6373/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 26
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 08:58:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6612/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 7
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 16 Jul 2020 05:13:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#28). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-1-5dbd44ad-18bc-40f2-9dd6-aeb2cc23457c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-3-27db2521-1e8b-40c1-b846-552cd620abce-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-4-f1b55628-0544-4833-8b11-1b4add53dfd6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-6-f75530ef-93b6-4994-b3c8-db957d44848c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-7-8d9b22da-5f10-4cbf-8e4d-160f829b5e48-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-9-f029a1f7-9024-4bc3-a030-e20861586146-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-11-f07814ae-56cd-486b-af81-18541437da7d-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-12-967c70a4-bf4d-4a82-8c97-c90e2b4d9dcf-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-14-d0cdca7f-c050-407e-b70c-2bd076f83e4e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-15-0e931a1f-309e-43b3-a5cf-3ef82fa4a87c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-17-43138078-244c-4b38-8127-04a5bfbc4695-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-19-52569895-df25-4ad8-b64d-49c4540d36c9-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-20-f160c1ea-a2f5-4109-81ec-3ff9c155430f-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-22-c1f61b8c-9d9a-4823-b64e-109770c16696-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-23-8cfc9d65-bfc3-47c4-8da7-6610b49b3305-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-25-d519b09d-c730-40a6-ad03-8b36096cb234-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-27-802b2129-0d18-418a-bcdd-b1b5332ea2a5-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-28-70f51a0c-3d30-4cdb-afcb-e8c1cfa3caa4-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-30-e887707d-58db-469c-ab96-b77188f25189-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-31-facf6a62-c326-44f7-bdc3-b1471bd594d7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-93996984692289973-1-9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-1-b975a171-0911-47c2-90c8-300f23c28772-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-6-9bc6f547-de52-4aad-8c07-5d0f1aaa0558-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-9-bf3024d6-8b55-42d3-b25e-5f2a31303026-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-17-a0d23269-c8d7-40cf-94c8-8b624969b9c0-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-23-138a376e-696e-4da3-95f3-7e0aacb11ca3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-27-04494175-75de-41ef-b824-3923ebfba0cb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-30-eba34050-b263-41e8-bc7d-773a47b33c32-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-31-0ec1c3ff-7154-4fd1-a090-8fa0b9c2d3c8-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-4-8b99da05-6be1-42e1-90e0-bff8ca08b5af-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-14-5b39012d-85e3-4144-ac83-574ed5046718-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-15-e37e3f98-5f3b-4245-9c01-76cf0f1dd21a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-19-e36a1955-f01c-4e08-9f19-dcec0cbceadd-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-20-c9632a29-fd5b-4ba5-a0db-2992c73075c2-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-28-31220e43-f85a-4609-9ebb-4c76269cddb6-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-3-7e47d440-44f5-4a11-bbfa-c72c520ca997-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-7-c902540c-b002-413c-b422-7ec3c06d404e-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-11-5558f0e5-7a46-417e-8c0a-50bb00e54a30-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-12-ea94b7c7-164d-4d06-8b4d-24d296906f73-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-22-0565b9eb-060d-4180-aa1e-2bb69187d8db-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-25-1f2b05c6-5407-41fc-a881-60626253ca77-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/af797bab-2f2c-44df-a77b-d91c7198fe53-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-8270633197658268308-1-af797bab-2f2c-44df-a77b-d91c7198fe53.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,442 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/28
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16143/20/testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
File testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json:
PS20:
> Hi Tim, do you mean that I need to use '$DEFAULT_FS' to replace 'hdfs://loc
DEFAULT_FS is an environment variables that's set in impala-config.sh E.g. on my dev environment it is:
DEFAULT_FS=hdfs://172.19.0.1:20500
And I get errors like this when loading data:
MESSAGE: TableLoadingException: Error loading metadata for Iceberg table hdfs://172.19.0.1:20500/test-warehouse/iceberg_test/iceberg_partitioned
CAUSED BY: RuntimeIOException: Failed to open input stream for file: hdfs://localhost:20500/test-warehouse/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
CAUSED BY: ConnectException: Call From tarmstrong-box2/127.0.0.1 to localhost:20500 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
CAUSED BY: ConnectException: Connection refused
With the JSON files we could do a textual search and replace as part of data loading, e.g. we could do something like this that I modified from testdata/bin/load-metastore-snapshot.sh
sed "s|hdfs://localhost:20500|${DEFAULT_FS}|g" testdata/data /iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json.template > testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
I tried doing that manually but I still had problems because the avro files seem to have hdfs://localhost:20500 hardcoded in them too. I can try to think of a solution but maybe you also have ideas.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sat, 29 Aug 2020 00:40:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 17:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6785/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 17
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 13:08:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
88 files changed, 1,539 insertions(+), 196 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/16
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 16
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 27:
(1 comment)
No worries, thanks for your persistence. Seems like something is wrong with test_iceberg.py in the dockerised test. I'm also trying to figure out what went wrong.
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/query_test.test_iceberg/TestCreatingIcebergTable/test_create_iceberg_tables_protocol__beeswax___exec_option____batch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0____table_format__parquet_none_/
http://gerrit.cloudera.org:8080/#/c/16143/26/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/16143/26/tests/query_test/test_iceberg.py@27
PS26, Line 27:
Similarly to test_scanners.py, we could constrain the file_format dimensions to Parquet.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 27
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 14:46:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 24: Code-Review+1
Thanks for the changes! I've just restarted the verify job on PS24.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 24
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 08:30:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
88 files changed, 1,510 insertions(+), 196 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/15
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 15
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 21:
Yeah, the verify job failed during loading the iceberg tables:
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3051/artifact/Impala/logs_static/logs/data_loading/sql/functional/invalidate-functional-query-exhaustive-impala-generated.sql.log
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 21
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 14:26:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 28:
Yeah, I was thinking about adding
@classmethod
def add_test_dimensions(cls):
super(TestCreatingIcebergTable, cls).add_test_dimensions()
cls.ImpalaTestMatrix.add_constraint(
lambda v: v.get_value('table_format').file_format == 'parquet')
to class TestCreatingIcebergTable. Since Parquet is kind of hard-coded to the tests, it has no additional value to run the tests with different file formats in the file format dimension.
Regarding to the build error, it was caused by IMPALA-9923 which is an unrelated issue.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 08:15:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#29). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-1-5dbd44ad-18bc-40f2-9dd6-aeb2cc23457c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-3-27db2521-1e8b-40c1-b846-552cd620abce-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-4-f1b55628-0544-4833-8b11-1b4add53dfd6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-6-f75530ef-93b6-4994-b3c8-db957d44848c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-7-8d9b22da-5f10-4cbf-8e4d-160f829b5e48-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-9-f029a1f7-9024-4bc3-a030-e20861586146-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-11-f07814ae-56cd-486b-af81-18541437da7d-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-12-967c70a4-bf4d-4a82-8c97-c90e2b4d9dcf-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-14-d0cdca7f-c050-407e-b70c-2bd076f83e4e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-15-0e931a1f-309e-43b3-a5cf-3ef82fa4a87c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-17-43138078-244c-4b38-8127-04a5bfbc4695-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-19-52569895-df25-4ad8-b64d-49c4540d36c9-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-20-f160c1ea-a2f5-4109-81ec-3ff9c155430f-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-22-c1f61b8c-9d9a-4823-b64e-109770c16696-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-23-8cfc9d65-bfc3-47c4-8da7-6610b49b3305-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-25-d519b09d-c730-40a6-ad03-8b36096cb234-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-27-802b2129-0d18-418a-bcdd-b1b5332ea2a5-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-28-70f51a0c-3d30-4cdb-afcb-e8c1cfa3caa4-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-30-e887707d-58db-469c-ab96-b77188f25189-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-31-facf6a62-c326-44f7-bdc3-b1471bd594d7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-93996984692289973-1-9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-1-b975a171-0911-47c2-90c8-300f23c28772-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-6-9bc6f547-de52-4aad-8c07-5d0f1aaa0558-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-9-bf3024d6-8b55-42d3-b25e-5f2a31303026-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-17-a0d23269-c8d7-40cf-94c8-8b624969b9c0-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-23-138a376e-696e-4da3-95f3-7e0aacb11ca3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-27-04494175-75de-41ef-b824-3923ebfba0cb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-30-eba34050-b263-41e8-bc7d-773a47b33c32-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-31-0ec1c3ff-7154-4fd1-a090-8fa0b9c2d3c8-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-4-8b99da05-6be1-42e1-90e0-bff8ca08b5af-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-14-5b39012d-85e3-4144-ac83-574ed5046718-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-15-e37e3f98-5f3b-4245-9c01-76cf0f1dd21a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-19-e36a1955-f01c-4e08-9f19-dcec0cbceadd-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-20-c9632a29-fd5b-4ba5-a0db-2992c73075c2-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-28-31220e43-f85a-4609-9ebb-4c76269cddb6-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-3-7e47d440-44f5-4a11-bbfa-c72c520ca997-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-7-c902540c-b002-413c-b422-7ec3c06d404e-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-11-5558f0e5-7a46-417e-8c0a-50bb00e54a30-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-12-ea94b7c7-164d-4d06-8b4d-24d296906f73-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-22-0565b9eb-060d-4180-aa1e-2bb69187d8db-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-25-1f2b05c6-5407-41fc-a881-60626253ca77-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/af797bab-2f2c-44df-a77b-d91c7198fe53-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-8270633197658268308-1-af797bab-2f2c-44df-a77b-d91c7198fe53.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,454 insertions(+), 203 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/29
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
Patch Set 1:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java@80
PS1, Line 80: "SHOW FILES not applicable to a non hdfs table and non iceberg table: %s", tableName_));
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java@84
PS1, Line 84: // There two cases here: Non-partitioned hdfs table and non-partitioned iceberg table
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@249
PS1, Line 249: public static Map<Integer, String> getPartitionColToSourceIdMap(List<IcebergPartitionSpec> specs) {
line too long (103 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@286
PS1, Line 286: tIcebergTable.setPartition_col_to_source_id_map(icebergTable.getPartitionColToSourceIdMap());
line too long (99 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@309
PS1, Line 309: private static HdfsPartition.FileDescriptor getFileDescriptor(FileSystem fs, Path tableLoc,
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@322
PS1, Line 322: return HdfsPartition.FileDescriptor.create(fileStatus, relPath, locations, hostIndex,
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@329
PS1, Line 329: public static Map<String, HdfsPartition.FileDescriptor> loadAllPartition(String location,
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@337
PS1, Line 337: HdfsPartition.FileDescriptor fileDesc = getFileDescriptor(new Path(file.path().toString()),
line too long (99 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@368
PS1, Line 368: partition.setFileFormat(IcebergUtil.toTHdfsFileFormat(icebergTable.getIcebergFileFormat()));
line too long (100 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java:
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java@176
PS1, Line 176: table_.getMetaStoreTable().getParameters().get(IcebergTable.ICEBERG_FILE_FORMAT);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@71
PS1, Line 71: Map<Integer, String> partitionColToSourceIdMap = Utils.getPartitionColToSourceIdMap(partitionSpecs);
line too long (104 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@78
PS1, Line 78: ColumnMap cmap, List<IcebergPartitionSpec> partitionSpecs, Map<String, Integer> sourceColsMap,
line too long (100 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@161
PS1, Line 161: localFsTable_.createPrototypePartition(), CatalogObject.ThriftObjectType.DESCRIPTOR_ONLY);
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@162
PS1, Line 162: THdfsTable hdfsTable = new THdfsTable(localFsTable_.getHdfsBaseDir(), getColumnNames(),
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@163
PS1, Line 163: localFsTable_.getNullPartitionKeyValue(), FeFsTable.DEFAULT_NULL_COLUMN_VALUE, idToPartition,
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@76
PS1, Line 76: * Get partition info from FeFsTable, we treat iceberg table as an unpartitioned hdfs table
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/16143/1/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1473
PS1, Line 1473: FeFsTable feFsTable = ((FeIcebergTable) tblRef.getDesc().getTable()).getFeFsTable();
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 1
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Jul 2020 07:21:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 19:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6886/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 19
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 12 Aug 2020 08:33:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 13:
(10 comments)
Thanks for applying the changes and for the tests. Unfortunately last time the gerrit "New UI" hid some files from me...
http://gerrit.cloudera.org:8080/#/c/16143/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16143/13//COMMIT_MSG@38
PS13, Line 38: custom cluster test test_iceberg.py
The tests are added to test_scanners.py
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/data/README
File testdata/data/README:
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/data/README@503
PS13, Line 503: Supported query icebreg table by impala
Please update this to reflect the current title
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/data/README@504
PS13, Line 504: Apache Iceberg is an open table format for huge analytic datasets.
: Iceberg avoids unpleasant surprises. Schema evolution works and won’t inadvertently
: un-delete data. Users don’t need to know about partitioning to get fast queries.
: And Iceberg was designed to solve correctness problems in eventually-consistent cloud
: object stores.
Please provide a description about the tables in the iceberg directory, not about Iceberg itself.
Also, please mention how the data was generated, e.g. what tool and what version.
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/datasets/functional/functional_schema_template.sql@2868
PS13, Line 2868: STORED AS ICEBERG
Don't you need to provide a partition spec?
Also, please add a SHOW PARTITIONS test for it in one of the test files.
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test:
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test@3
PS13, Line 3: Supported query icebreg table by impala
Please update this
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test@42
PS13, Line 42: 14
You could add a RUNTIME_PROFILE section to check that partition pruning worked. E.g.
https://github.com/apache/impala/blob/76e4a17fb379bb232618dccb4ad3504dbe5c945c/testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test#L29
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test@77
PS13, Line 77: SELECT count(*) from iceberg_non_partitioned
Please add non-count queries as well.
After you are convinced that the queries produce the correct results, you can use the following command to re-generate this file:
bin/impala-py.test --update_results tests/query_test/test_scanners.py::TestIceberg::test_iceberg
You'll find the generated .test file in 'logs/ee_tests/'
Alternatively, you can set these in the Impala shell:
set write_delimited=true;
set delimiter=,;
Then Impala shell will produce an output that can be copy-pasted here.
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test@83
PS13, Line 83: ====
Please add SHOW FILES for each table.
Also, you could add an iceberg-negative.test that'd check we get proper error messages for features not supported for Iceberg. E.g.: https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/acid-negative.test
http://gerrit.cloudera.org:8080/#/c/16143/13/tests/custom_cluster/test_iceberg.py
File tests/custom_cluster/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/16143/13/tests/custom_cluster/test_iceberg.py@23
PS13, Line 23: class TestCreatingIcebergTable(CustomClusterTestSuite):
This test could be put into tests/query_test, instead of tests/custom_cluster. And it could just inherit from ImpalaTestSuite. This way we wouldn't need to restart the Impala cluster whenever this test is being run.
http://gerrit.cloudera.org:8080/#/c/16143/13/tests/custom_cluster/test_iceberg.py@30
PS13, Line 30: @pytest.mark.execute_serially
I think there's no need for this annotation. Since this test is using a unique database it can be run in parallel with other tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sun, 19 Jul 2020 11:34:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 18:
All test cases passed for patch 18: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11604/
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 18
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sun, 09 Aug 2020 01:49:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6496/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 2
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Jul 2020 08:01:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 25: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6366/
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 16:20:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6887/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 12 Aug 2020 08:39:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
88 files changed, 1,510 insertions(+), 194 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/14
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 14
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 13:
Test passed, and iceberg test worked in query_test.test_scanners, here is the Jenkins url: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11338
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sat, 18 Jul 2020 01:36:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16143/6/testdata/bin/generate-schema-statements.py
File testdata/bin/generate-schema-statements.py:
http://gerrit.cloudera.org:8080/#/c/16143/6/testdata/bin/generate-schema-statements.py@766
PS6, Line 766: n
flake8: E501 line too long (94 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16143/6/tests/common/test_dimensions.py
File tests/common/test_dimensions.py:
http://gerrit.cloudera.org:8080/#/c/16143/6/tests/common/test_dimensions.py@32
PS6, Line 32: c
flake8: E501 line too long (98 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 6
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 15 Jul 2020 08:05:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 23:
Hi Tim,Zoltan, thanks for your review. I've been discussed with my colleague who is research Iceberg. And we found that we can generate iceberg data without hardcoded 'hdfs://localhost:20500' in metadata, so we regenerated these test files, and update patch. Now these json and avro files do not contain 'hdfs://localhost:20500'. Would you please restart job to verify the patch?
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 23
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 06:28:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
Patch Set 4:
(10 comments)
Thanks for working on this, it will be a really great addition to Impala!
http://gerrit.cloudera.org:8080/#/c/16143/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16143/4//COMMIT_MSG@7
PS4, Line 7: icebreg
Iceberg
http://gerrit.cloudera.org:8080/#/c/16143/4//COMMIT_MSG@26
PS4, Line 26: PARQUT
PARQUET
http://gerrit.cloudera.org:8080/#/c/16143/4//COMMIT_MSG@27
PS4, Line 27:
Please add a high-level description about what this patch does.
http://gerrit.cloudera.org:8080/#/c/16143/4/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/16143/4/common/thrift/CatalogObjects.thrift@510
PS4, Line 510: source_cols_map
please add some comment about the fields
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java@86
PS4, Line 86: boolean flag = op_ == TShowStatsOp.PARTITIONS ? table_ instanceof FeIcebergTable ?
: ((FeIcebergTable) table_).getPartitionColToSourceIdMap().isEmpty() :
: table_.getNumClusteringCols() == 0 : false;
nit: for readability, please use if statements instead of nested ternary operators
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@69
PS4, Line 69: transfromed
transformed
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@71
PS4, Line 71: getFeFsTable
Maybe rename to getHdfsTable() ?
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@115
PS4, Line 115: toUpperCase();
: if (format.equalsIgnoreCase
nit: toUpperCase() or equalsIgnoreCase() is not needed.
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@246
PS4, Line 246: List<DataFile> dataFileList = new ArrayList<>();
: for (FileScanTask task : scan.planFiles()) {
: dataFileList.add(task.file());
: }
: return dataFileList;
nit: return Lists.newArrayList(scan.planFiles());
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/test/java/org/apache/impala/customcluster/IcebergTableQueryTest.java
File fe/src/test/java/org/apache/impala/customcluster/IcebergTableQueryTest.java:
http://gerrit.cloudera.org:8080/#/c/16143/4/fe/src/test/java/org/apache/impala/customcluster/IcebergTableQueryTest.java@42
PS4, Line 42: /**
: * Test impala query iceberg table
: * impala not supported insert into iceberg table now, so we construct iceberg
: * table by iceberg api
: */
Instead of writing the Iceberg table each time, can we just check it into the repository then copy it to the HDFS warehouse directory during data loading?
We did something similar with Apache Hudi: https://gerrit.cloudera.org/c/14711/
After that you could create end-to-end tests in Python and in ".test" files. E.g.:
https://github.com/apache/impala/blob/65722d3e9051d6a08cb1e69fd36a06684745c226/tests/query_test/test_scanners.py#L326-L340
https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/hudi-parquet.test
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 4
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2020 14:09:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 24:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6364/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 24
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 08:30:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 28:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7071/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 03:41:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6549/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 4
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2020 12:54:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 14:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16143/14/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/16143/14/tests/query_test/test_iceberg.py@18
PS14, Line 18: import pytest
flake8: F401 'pytest' imported but unused
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 14
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 20 Jul 2020 11:27:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,436 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/21
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 21
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/query_test/test_scanners.py
82 files changed, 1,311 insertions(+), 161 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/13
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6351/
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 01:51:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 26:
FileMetadataLoaderTest.testIcebergLoading and test_create_iceberg_tables still fail:
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11899/testReport/
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 26
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 13:34:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 30:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7116/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 30
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sat, 05 Sep 2020 02:55:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
IMPALA-9741: Support quering Icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When query iceberg table, we pushdown
partition column predicates to iceberg to decided which data files
need to be scanned, and then transformed these information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/query_test/test_scanners.py
82 files changed, 1,310 insertions(+), 161 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/12
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 12
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6604/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 6
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 15 Jul 2020 08:32:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
Patch Set 8:
(13 comments)
Done!
http://gerrit.cloudera.org:8080/#/c/16143/6/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/16143/6/common/thrift/CatalogObjects.thrift@512
PS6, Line 512: column_to_sourc
> nit: column_to_source_id ?
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/common/thrift/CatalogObjects.thrift@515
PS6, Line 515: source_id_to_partition
> The mapping is reversed. Name it "source_id_to_partition" ?
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/common/thrift/CatalogObjects.thrift@516
PS6, Line 516: map<string,THdfsFileDesc> path_md5_to_file
> Please follow the above conventions for naming maps.
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
File fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java:
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java@28
PS6, Line 28: // The id of the source column in the Iceberg table schema. The source column is
: // used as the input for this partition field.
> Might worth rewording it a bit:
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java@88
PS6, Line 88: if (table_ instanceof FeIcebergTable) {
: if (((FeIcebergTable) table_).getSourceIdToPartitionMap().isEmpty()) {
: notPartitioned = true;
: }
> Probably we should treat all Iceberg tables as not partitioned, since it's
Yes, you are right, we treated iceberg table as unpartitioned hdfs table, but iceberg table still has it's own partition info, we get this info by 'show partitions xxx.iceberg_table_test' like this:
+--------------+-----------+----------+------------+---------------------------+
| Partition Id | Source Id | Field Id | Field Name | Field Partition Transform |
+--------------+-----------+----------+------------+---------------------------+
| 0 | 2 | 1000 | sex | IDENTITY |
| 0 | 3 | 1001 | action | IDENTITY |
+--------------+-----------+----------+------------+---------------------------+
If I set 'notPartitioned' as true, even if getPartitionColToSourceIdMap() is not empty, how can I get the iceberg partition info? 'show partitions xxx.iceberg_table_test' will always return AnalysisException.
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@66
PS6, Line 66: getPathMD5ToFi
> nit: getPartitionToFileDescMap
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@219
PS6, Line 219: isPartitioned(Fe
> nit: isPartitioned?
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@258
PS6, Line 258: PartitionColToSourceId
> It returns a mapping from source ids to partition columns, therefore please
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@271
PS6, Line 271: getColumnToSourc
> nit: getColumnToSourceIdMap?
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@305
PS6, Line 305:
> nit: wrong indentation
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@114
PS6, Line 114: if ("PARQUET".equalsIgnoreCase(format)) return TIcebergFileFormat.PARQUET;
: return null;
: }
:
: /**
: * Build TIceb
> How about:
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/testdata/bin/generate-schema-statements.py
File testdata/bin/generate-schema-statements.py:
http://gerrit.cloudera.org:8080/#/c/16143/6/testdata/bin/generate-schema-statements.py@193
PS6, Line 193: }
> You probably don't need to modify this file. I think adding HUDIPARQUET to
Done
http://gerrit.cloudera.org:8080/#/c/16143/6/testdata/bin/generate-schema-statements.py@766
PS6, Line 766:
> flake8: E501 line too long (94 > 90 characters)
Done
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 8
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 16 Jul 2020 06:26:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 30:
(2 comments)
> Thanks for fixing the test issues. The PS needs to be rebased to
> master.
> I can give it +2 if it's OK to Tim.
Thanks, Zoltan, already rebase to latest commit, and Jenkins test passed: https://jenkins.impala.io/job/pre-review-test/681/
http://gerrit.cloudera.org:8080/#/c/16143/28/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/16143/28/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3199
PS28, Line 3199: ((IcebergTable) table).getHdfsTable();
> Here we return IcebergTable.hdfsTable_, which might be obsolete. See Iceber
Done
http://gerrit.cloudera.org:8080/#/c/16143/28/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/28/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@185
PS28, Line 185: // Copy the table to check later if anything has changed.
> We load 'hdfsTable_' here, however the schema might be updated at L192. Bec
Done
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 30
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sat, 05 Sep 2020 15:09:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16143/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@208
PS9, Line 208: pathMD5ToFileDescMap_ = Utils.loadAllPartition(msTable_.getSd().getLocation(), this);
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@277
PS9, Line 277: pathMD5ToFileDescMap_ = loadFileDescFromThrift(ticeberg.getPath_md5_to_file_descriptors());
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/9/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/9/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@102
PS9, Line 102: icebergTable_.getPathMD5ToFileDescMap().get(IcebergUtil.getDataFileMD5(dataFile));
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 9
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 16 Jul 2020 06:29:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
88 files changed, 1,539 insertions(+), 197 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/18
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 18
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 21:
> (1 comment)
Hi Tim, thanks for your patient review and advice, I understand your meaning. I will also try to think about solutions about the hardcoded 'hdfs://localhost:20500' in avro files. But still a question: how to get DEFAULT_FS in different environment? from core-site.xml? Maybe a stupid question, but I'm not familiar with other storage system besides HDFS.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 21
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sat, 29 Aug 2020 08:30:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 16:
Test passed, here is the Jenkins url for patch 16: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11367/
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 16
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 22 Jul 2020 01:32:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 15:
Oh and please do a rebase on top of master because gerrit says that this PS currently has a merge conflict.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 15
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 20 Jul 2020 17:46:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,437 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/20
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#30). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-1-5dbd44ad-18bc-40f2-9dd6-aeb2cc23457c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-3-27db2521-1e8b-40c1-b846-552cd620abce-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-4-f1b55628-0544-4833-8b11-1b4add53dfd6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-6-f75530ef-93b6-4994-b3c8-db957d44848c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-7-8d9b22da-5f10-4cbf-8e4d-160f829b5e48-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-9-f029a1f7-9024-4bc3-a030-e20861586146-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-11-f07814ae-56cd-486b-af81-18541437da7d-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-12-967c70a4-bf4d-4a82-8c97-c90e2b4d9dcf-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-14-d0cdca7f-c050-407e-b70c-2bd076f83e4e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-15-0e931a1f-309e-43b3-a5cf-3ef82fa4a87c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-17-43138078-244c-4b38-8127-04a5bfbc4695-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-19-52569895-df25-4ad8-b64d-49c4540d36c9-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-20-f160c1ea-a2f5-4109-81ec-3ff9c155430f-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-22-c1f61b8c-9d9a-4823-b64e-109770c16696-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-23-8cfc9d65-bfc3-47c4-8da7-6610b49b3305-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-25-d519b09d-c730-40a6-ad03-8b36096cb234-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-27-802b2129-0d18-418a-bcdd-b1b5332ea2a5-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-28-70f51a0c-3d30-4cdb-afcb-e8c1cfa3caa4-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-30-e887707d-58db-469c-ab96-b77188f25189-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-31-facf6a62-c326-44f7-bdc3-b1471bd594d7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-93996984692289973-1-9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-1-b975a171-0911-47c2-90c8-300f23c28772-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-6-9bc6f547-de52-4aad-8c07-5d0f1aaa0558-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-9-bf3024d6-8b55-42d3-b25e-5f2a31303026-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-17-a0d23269-c8d7-40cf-94c8-8b624969b9c0-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-23-138a376e-696e-4da3-95f3-7e0aacb11ca3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-27-04494175-75de-41ef-b824-3923ebfba0cb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-30-eba34050-b263-41e8-bc7d-773a47b33c32-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-31-0ec1c3ff-7154-4fd1-a090-8fa0b9c2d3c8-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-4-8b99da05-6be1-42e1-90e0-bff8ca08b5af-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-14-5b39012d-85e3-4144-ac83-574ed5046718-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-15-e37e3f98-5f3b-4245-9c01-76cf0f1dd21a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-19-e36a1955-f01c-4e08-9f19-dcec0cbceadd-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-20-c9632a29-fd5b-4ba5-a0db-2992c73075c2-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-28-31220e43-f85a-4609-9ebb-4c76269cddb6-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-3-7e47d440-44f5-4a11-bbfa-c72c520ca997-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-7-c902540c-b002-413c-b422-7ec3c06d404e-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-11-5558f0e5-7a46-417e-8c0a-50bb00e54a30-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-12-ea94b7c7-164d-4d06-8b4d-24d296906f73-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-22-0565b9eb-060d-4180-aa1e-2bb69187d8db-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-25-1f2b05c6-5407-41fc-a881-60626253ca77-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/af797bab-2f2c-44df-a77b-d91c7198fe53-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-8270633197658268308-1-af797bab-2f2c-44df-a77b-d91c7198fe53.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,454 insertions(+), 203 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/30
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 30
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20: Code-Review+1
(4 comments)
Did another pass, found a few nits, but the code LGTM. Only giving it +1 because Tim mentioned he'll do another round as well.
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java@86
PS20, Line 86: notPartitioned = false;
nit: probably just call this variable 'partitioned', having a negation in a variable name is confusing a bit.
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@134
PS20, Line 134: // LocalFsTable transformed from iceberg table only has one partition
nit: then maybe we could use a Preconditions check to verify this. Also, we don't need a for-loop in this case.
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@117
PS20, Line 117: 1
In case 1 we cannot filter data files because the column is not part of the partition expressions, right?
http://gerrit.cloudera.org:8080/#/c/16143/20/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test:
http://gerrit.cloudera.org:8080/#/c/16143/20/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test@4
PS20, Line 4: USE functional_parquet;
nit: it can be omitted
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 27 Aug 2020 15:11:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Anonymous Coward (606), Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16143
to look at the new patch set (#17).
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
88 files changed, 1,538 insertions(+), 197 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/17
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 17
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 30: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 30
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sat, 05 Sep 2020 20:54:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6677/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 16
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 21 Jul 2020 07:09:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 16:
(6 comments)
Hi Zoltan, I've already modify code and release patch to the latest commit. I will submit a Jenkins test again to ensure everything right.
http://gerrit.cloudera.org:8080/#/c/16143/15/fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
File fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/15/fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java@66
PS15, Line 66: we also cannot truncate Iceberg
> nit: "we also cannot truncate Iceberg tables."
Done
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README
File testdata/data/README:
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@503
PS15, Line 503: t has s
> nit: Iceberg
Done
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@507
PS15, Line 507:
> managed
Done
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@508
PS15, Line 508:
> nit: which contains the data files.
Done
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test:
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test@19
PS15, Line 19: AnalysisException: Cannot find source column: event_time
: ====
> Can we catch this error during analysis, and throw an AnalysisException ins
Done
This exception throws from iceberg, I didn't check this in impala. Since you have mentioned this question, I add checkPartitionColumns in CreateTableStmt.java for iceberg managed table.
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test:
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test@7
PS15, Line 7: aggregation(SUM, RowsRead): 6
: ====
> I'm just wondering how deterministic is it. Do the test consistently pass i
Done
use 'aggregation(SUM, RowsRead)' to replace 'row_regex: .* PARQUET/GZIP'
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 16
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 21 Jul 2020 06:42:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6614/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 9
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 16 Jul 2020 06:54:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 24:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7050/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 24
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 08:45:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
IMPALA-9741: Supported query icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently support
PARQUET and ORC. And if you don't identity this property in your SQL,
default file format is PARQUT.
Testing:
- Add fe test IcebergTableQueryTest.java
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A fe/src/test/java/org/apache/impala/customcluster/IcebergTableQueryTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
25 files changed, 1,148 insertions(+), 158 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/4
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 4
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 13:
(8 comments)
Hi Zoltan, thanks for your review again. I've already modify code and submit test case on Jenkins for patch 13. I will update here when all test cases passed and iceberg query test case worked.
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@7
PS11, Line 7: querying Iceber
> misspelled querying
Done
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@9
PS11, Line 9: query
> nit: querying
Done
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@30
PS11, Line 30: query
> querying
Done
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@31
PS11, Line 31: decide
> decide
Done
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@32
PS11, Line 32: transfer this inf
> transfer this
Done
http://gerrit.cloudera.org:8080/#/c/16143/11/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/16143/11/common/thrift/CatalogObjects.thrift@516
PS11, Line 516:
> nit: Plural 's' not needed since the value type of the map is not a list
Done
http://gerrit.cloudera.org:8080/#/c/16143/11/common/thrift/CatalogObjects.thrift@516
PS11, Line 516: // Data file path md5 and it's file descriptor
> nit: please add comment about field 5.
Done
http://gerrit.cloudera.org:8080/#/c/16143/11/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/11/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@258
PS11, Line 258: SourceIdToPartitionCol
> SourceIdToPartition
Done
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 17 Jul 2020 12:38:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
IMPALA-9741: Support quering Icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When query iceberg table, we pushdown
partition column predicates to iceberg to decided which data files
need to be scanned, and then transformed these information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/custom_cluster/test_iceberg.py
M tests/query_test/test_scanners.py
83 files changed, 1,322 insertions(+), 161 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/10
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 10
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
Patch Set 8:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16143/8/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/8/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@208
PS8, Line 208: pathMD5ToFileDescMap_ = Utils.loadAllPartition(msTable_.getSd().getLocation(), this);
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/8/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@277
PS8, Line 277: pathMD5ToFileDescMap_ = loadFileDescFromThrift(ticeberg.getPath_md5_to_file_descriptors());
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/8/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/8/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@102
PS8, Line 102: icebergTable_.getPathMD5ToFileDescMap().get(IcebergUtil.getDataFileMD5(dataFile));
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 8
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 16 Jul 2020 06:28:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
IMPALA-9741: Support quering Icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When query iceberg table, we pushdown
partition column predicates to iceberg to decided which data files
need to be scanned, and then transformed these information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/query_test/test_scanners.py
82 files changed, 1,310 insertions(+), 161 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/11
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 11
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 24: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6364/
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 24
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 09:44:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 29: Code-Review+1
Thanks for fixing the test issues. The PS needs to be rebased to master.
I can give it +2 if it's OK to Tim.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 04 Sep 2020 14:05:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
IMPALA-9741: Supported query icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently support
PARQUET and ORC. And if you don't identity this property in your SQL,
default file format is PARQUT.
Testing:
- Add fe test IcebergTableQueryTest.java
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A fe/src/test/java/org/apache/impala/customcluster/IcebergTableQueryTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
25 files changed, 1,082 insertions(+), 155 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/3
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 3
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Anonymous Coward (606) has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 16:
(4 comments)
The code looks fine. All of my comments are optional. Thank you for the opportunity to review.
http://gerrit.cloudera.org:8080/#/c/16143/16//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16143/16//COMMIT_MSG@29
PS16, Line 29: We achieved this function by treating the iceberg table as normal
Are there plans to support read of Iceberg table as a partitioned table? This will help with collocated joins.
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java@80
PS16, Line 80: "SHOW FILES not applicable to a non hdfs table and non iceberg table: %s",
The double negative is pretty hard to parse.
How about:
"SHOW FILES is applicable only to a HDFS table"?
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@139
PS16, Line 139: if (!(predicate.getChild(0) instanceof SlotRef)) return false;
Can the predicate be of the form: '10 = p1'? In that case, should there be symmetric code to get slotRef and literalExpr?
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@114
PS16, Line 114: if ("PARQUET".equalsIgnoreCase(format)) return TIcebergFileFormat.PARQUET;
Rest of the code seems to support Iceberg ORC file format. This code does not seem to support it?
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 16
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 03 Aug 2020 16:46:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Reviewed-on: http://gerrit.cloudera.org:8080/16143
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-1-5dbd44ad-18bc-40f2-9dd6-aeb2cc23457c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-3-27db2521-1e8b-40c1-b846-552cd620abce-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-4-f1b55628-0544-4833-8b11-1b4add53dfd6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-6-f75530ef-93b6-4994-b3c8-db957d44848c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-7-8d9b22da-5f10-4cbf-8e4d-160f829b5e48-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-9-f029a1f7-9024-4bc3-a030-e20861586146-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-11-f07814ae-56cd-486b-af81-18541437da7d-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-12-967c70a4-bf4d-4a82-8c97-c90e2b4d9dcf-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-14-d0cdca7f-c050-407e-b70c-2bd076f83e4e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-15-0e931a1f-309e-43b3-a5cf-3ef82fa4a87c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-17-43138078-244c-4b38-8127-04a5bfbc4695-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-19-52569895-df25-4ad8-b64d-49c4540d36c9-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-20-f160c1ea-a2f5-4109-81ec-3ff9c155430f-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-22-c1f61b8c-9d9a-4823-b64e-109770c16696-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-23-8cfc9d65-bfc3-47c4-8da7-6610b49b3305-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-25-d519b09d-c730-40a6-ad03-8b36096cb234-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-27-802b2129-0d18-418a-bcdd-b1b5332ea2a5-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-28-70f51a0c-3d30-4cdb-afcb-e8c1cfa3caa4-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-30-e887707d-58db-469c-ab96-b77188f25189-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-31-facf6a62-c326-44f7-bdc3-b1471bd594d7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-93996984692289973-1-9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-1-b975a171-0911-47c2-90c8-300f23c28772-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-6-9bc6f547-de52-4aad-8c07-5d0f1aaa0558-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-9-bf3024d6-8b55-42d3-b25e-5f2a31303026-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-17-a0d23269-c8d7-40cf-94c8-8b624969b9c0-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-23-138a376e-696e-4da3-95f3-7e0aacb11ca3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-27-04494175-75de-41ef-b824-3923ebfba0cb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-30-eba34050-b263-41e8-bc7d-773a47b33c32-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-31-0ec1c3ff-7154-4fd1-a090-8fa0b9c2d3c8-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-4-8b99da05-6be1-42e1-90e0-bff8ca08b5af-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-14-5b39012d-85e3-4144-ac83-574ed5046718-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-15-e37e3f98-5f3b-4245-9c01-76cf0f1dd21a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-19-e36a1955-f01c-4e08-9f19-dcec0cbceadd-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-20-c9632a29-fd5b-4ba5-a0db-2992c73075c2-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-28-31220e43-f85a-4609-9ebb-4c76269cddb6-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-3-7e47d440-44f5-4a11-bbfa-c72c520ca997-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-7-c902540c-b002-413c-b422-7ec3c06d404e-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-11-5558f0e5-7a46-417e-8c0a-50bb00e54a30-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-12-ea94b7c7-164d-4d06-8b4d-24d296906f73-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-22-0565b9eb-060d-4180-aa1e-2bb69187d8db-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-25-1f2b05c6-5407-41fc-a881-60626253ca77-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/af797bab-2f2c-44df-a77b-d91c7198fe53-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-8270633197658268308-1-af797bab-2f2c-44df-a77b-d91c7198fe53.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,454 insertions(+), 203 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 32
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6495/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 1
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Jul 2020 07:49:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,437 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/19
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 19
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20:
Thanks for the update - I'll do another pass over the code tomorrow.
I think we can probably do more optimisations for partition pruning later, but this will be simpler for now.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 03:25:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6632/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 12
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 17 Jul 2020 12:35:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
Patch Set 6:
(15 comments)
http://gerrit.cloudera.org:8080/#/c/16143/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16143/6//COMMIT_MSG@7
PS6, Line 7: Icebreg
It's still misspelled
http://gerrit.cloudera.org:8080/#/c/16143/6//COMMIT_MSG@7
PS6, Line 7: Supported query
nit: Support querying
http://gerrit.cloudera.org:8080/#/c/16143/6//COMMIT_MSG@26
PS6, Line 26: identity
specify
http://gerrit.cloudera.org:8080/#/c/16143/6/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/16143/6/common/thrift/CatalogObjects.thrift@512
PS6, Line 512: source_cols_map
nit: column_to_source_id ?
http://gerrit.cloudera.org:8080/#/c/16143/6/common/thrift/CatalogObjects.thrift@515
PS6, Line 515: partition_col_to_source_id_map
The mapping is reversed. Name it "source_id_to_partition" ?
http://gerrit.cloudera.org:8080/#/c/16143/6/common/thrift/CatalogObjects.thrift@516
PS6, Line 516: map<string,THdfsFileDesc> file_descriptors
Please follow the above conventions for naming maps.
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
File fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java:
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java@28
PS6, Line 28: // The id of the source field in iceberg table Schema, you can get these source
: // fields by Schema.columns(), the return type is List<NestedField>.
Might worth rewording it a bit:
"The id of the source column in the Iceberg table schema. The source column is used as the input for this partition field."
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java@88
PS6, Line 88: if (table_ instanceof FeIcebergTable) {
: if (((FeIcebergTable) table_).getPartitionColToSourceIdMap().isEmpty()) {
: notPartitioned = true;
: }
Probably we should treat all Iceberg tables as not partitioned, since it's partitioning is different than other file system tables' partitioning.
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@66
PS6, Line 66: getFileDescMap
nit: getPartitionToFileDescMap
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@219
PS6, Line 219: isPartitionTable
nit: isPartitioned?
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@258
PS6, Line 258: PartitionColToSourceId
It returns a mapping from source ids to partition columns, therefore please name it "sourceIdToPartitionCol".
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@271
PS6, Line 271: getSourceColsMap
nit: getColumnToSourceIdMap?
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@305
PS6, Line 305:
nit: wrong indentation
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@114
PS6, Line 114: if (format == null) return null;
: format = format.toUpperCase();
: if (format.equals("PARQUET")) {
: return TIcebergFileFormat.PARQUET;
: }
: return null;
How about:
if ("PARQUET".equalsIgnoreCase(format)) return TIcebergFileFormat.PARQUET;
return null;
http://gerrit.cloudera.org:8080/#/c/16143/6/testdata/bin/generate-schema-statements.py
File testdata/bin/generate-schema-statements.py:
http://gerrit.cloudera.org:8080/#/c/16143/6/testdata/bin/generate-schema-statements.py@193
PS6, Line 193: 'iceberg': 'ICEBERG'
You probably don't need to modify this file. I think adding HUDIPARQUET to this file was also unnecessary.
Probably we can do the same thing that we did for Hudi, i.e. add the Iceberg tables under the functional_parquet database.
https://gerrit.cloudera.org/c/14711/25/testdata/datasets/functional/schema_constraints.csv
https://gerrit.cloudera.org/c/14711/25/testdata/datasets/functional/functional_schema_template.sql
After that you should be able to load the Iceberg tables via Impala's data loading facilities (e.g. bin/load-data.py), and write some tests, like:
https://gerrit.cloudera.org/c/14711/25/tests/query_test/test_scanners.py
https://gerrit.cloudera.org/c/14711/25/testdata/workloads/functional-query/queries/QueryTest/hudi-parquet.test
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 6
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 15 Jul 2020 15:01:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 26: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6373/
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 26
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 14:12:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6593/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 5
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 14 Jul 2020 15:18:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20:
The GVO dryrun job failed by IMPALA-9923, restarted the job.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 09:16:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
Patch Set 8:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16143/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16143/6//COMMIT_MSG@7
PS6, Line 7: Icebreg
> It's still misspelled
Done
http://gerrit.cloudera.org:8080/#/c/16143/6//COMMIT_MSG@26
PS6, Line 26: identity
> specify
Done
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 8
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 16 Jul 2020 06:28:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 16: Code-Review+1
Thank you for your great work and patience! LGTM!
I give it +1 and ask some people to also take a look.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 16
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 21 Jul 2020 09:51:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 27:
> (1 comment)
>
> No worries, thanks for your persistence. Seems like something is
> wrong with test_iceberg.py in the dockerised test. I'm also trying
> to figure out what went wrong.
>
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/query_test.test_iceberg/TestCreatingIcebergTable/test_create_iceberg_tables_protocol__beeswax___exec_option____batch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0____table_format__parquet_none_/
Hi Zoltan, here is the test I submitted on Jenkins according to patch 27: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11905/.
Also failed, but seems unrelated to icberg: "23:12:14 15:12:14 Error executing hive SQL: /home/ubuntu/Impala/logs/data_loading/sql/tpcds/load-tpcds-core-hive-generated-orc-def-block.sql See: /home/ubuntu/Impala/logs/data_loading/sql/tpcds/load-tpcds-core-hive-generated-orc-def-block.sql.log".
I've download the log, found the reason: "Error while compiling statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checksum, since file /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451163/base_0000002/_orc_acid_version is under construction" I'm not sure is this error caused by other patch?
Besides, you mentioned "Similarly to test_scanners.py, we could constrain the file_format dimensions to Parquet.", do you mean like this?
@classmethod
def add_test_dimensions(cls):
super(TestHudiParquet, cls).add_test_dimensions()
cls.ImpalaTestMatrix.add_dimension(
create_exec_option_dimension(debug_action_options=DEBUG_ACTION_DIMS))
cls.ImpalaTestMatrix.add_constraint(
lambda v: v.get_value('table_format').file_format == 'parquet')
To be honest, I'm not very faimilar about this...
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 27
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 02:23:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6633/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 17 Jul 2020 13:01:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#26). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-1-5dbd44ad-18bc-40f2-9dd6-aeb2cc23457c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-3-27db2521-1e8b-40c1-b846-552cd620abce-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-4-f1b55628-0544-4833-8b11-1b4add53dfd6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-6-f75530ef-93b6-4994-b3c8-db957d44848c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-7-8d9b22da-5f10-4cbf-8e4d-160f829b5e48-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-9-f029a1f7-9024-4bc3-a030-e20861586146-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-11-f07814ae-56cd-486b-af81-18541437da7d-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-12-967c70a4-bf4d-4a82-8c97-c90e2b4d9dcf-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-14-d0cdca7f-c050-407e-b70c-2bd076f83e4e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-15-0e931a1f-309e-43b3-a5cf-3ef82fa4a87c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-17-43138078-244c-4b38-8127-04a5bfbc4695-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-19-52569895-df25-4ad8-b64d-49c4540d36c9-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-20-f160c1ea-a2f5-4109-81ec-3ff9c155430f-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-22-c1f61b8c-9d9a-4823-b64e-109770c16696-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-23-8cfc9d65-bfc3-47c4-8da7-6610b49b3305-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-25-d519b09d-c730-40a6-ad03-8b36096cb234-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-27-802b2129-0d18-418a-bcdd-b1b5332ea2a5-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-28-70f51a0c-3d30-4cdb-afcb-e8c1cfa3caa4-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-30-e887707d-58db-469c-ab96-b77188f25189-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-31-facf6a62-c326-44f7-bdc3-b1471bd594d7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-93996984692289973-1-9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-1-b975a171-0911-47c2-90c8-300f23c28772-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-6-9bc6f547-de52-4aad-8c07-5d0f1aaa0558-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-9-bf3024d6-8b55-42d3-b25e-5f2a31303026-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-17-a0d23269-c8d7-40cf-94c8-8b624969b9c0-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-23-138a376e-696e-4da3-95f3-7e0aacb11ca3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-27-04494175-75de-41ef-b824-3923ebfba0cb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-30-eba34050-b263-41e8-bc7d-773a47b33c32-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-31-0ec1c3ff-7154-4fd1-a090-8fa0b9c2d3c8-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-4-8b99da05-6be1-42e1-90e0-bff8ca08b5af-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-14-5b39012d-85e3-4144-ac83-574ed5046718-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-15-e37e3f98-5f3b-4245-9c01-76cf0f1dd21a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-19-e36a1955-f01c-4e08-9f19-dcec0cbceadd-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-20-c9632a29-fd5b-4ba5-a0db-2992c73075c2-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-28-31220e43-f85a-4609-9ebb-4c76269cddb6-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-3-7e47d440-44f5-4a11-bbfa-c72c520ca997-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-7-c902540c-b002-413c-b422-7ec3c06d404e-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-11-5558f0e5-7a46-417e-8c0a-50bb00e54a30-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-12-ea94b7c7-164d-4d06-8b4d-24d296906f73-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-22-0565b9eb-060d-4180-aa1e-2bb69187d8db-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-25-1f2b05c6-5407-41fc-a881-60626253ca77-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/af797bab-2f2c-44df-a77b-d91c7198fe53-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-8270633197658268308-1-af797bab-2f2c-44df-a77b-d91c7198fe53.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,436 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/26
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 26
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#27). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-1-5dbd44ad-18bc-40f2-9dd6-aeb2cc23457c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-3-27db2521-1e8b-40c1-b846-552cd620abce-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-4-f1b55628-0544-4833-8b11-1b4add53dfd6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-6-f75530ef-93b6-4994-b3c8-db957d44848c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-7-8d9b22da-5f10-4cbf-8e4d-160f829b5e48-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-9-f029a1f7-9024-4bc3-a030-e20861586146-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-11-f07814ae-56cd-486b-af81-18541437da7d-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-12-967c70a4-bf4d-4a82-8c97-c90e2b4d9dcf-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-14-d0cdca7f-c050-407e-b70c-2bd076f83e4e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-15-0e931a1f-309e-43b3-a5cf-3ef82fa4a87c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-17-43138078-244c-4b38-8127-04a5bfbc4695-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-19-52569895-df25-4ad8-b64d-49c4540d36c9-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-20-f160c1ea-a2f5-4109-81ec-3ff9c155430f-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-22-c1f61b8c-9d9a-4823-b64e-109770c16696-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-23-8cfc9d65-bfc3-47c4-8da7-6610b49b3305-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-25-d519b09d-c730-40a6-ad03-8b36096cb234-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-27-802b2129-0d18-418a-bcdd-b1b5332ea2a5-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-28-70f51a0c-3d30-4cdb-afcb-e8c1cfa3caa4-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-30-e887707d-58db-469c-ab96-b77188f25189-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-31-facf6a62-c326-44f7-bdc3-b1471bd594d7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-93996984692289973-1-9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-1-b975a171-0911-47c2-90c8-300f23c28772-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-6-9bc6f547-de52-4aad-8c07-5d0f1aaa0558-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-9-bf3024d6-8b55-42d3-b25e-5f2a31303026-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-17-a0d23269-c8d7-40cf-94c8-8b624969b9c0-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-23-138a376e-696e-4da3-95f3-7e0aacb11ca3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-27-04494175-75de-41ef-b824-3923ebfba0cb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-30-eba34050-b263-41e8-bc7d-773a47b33c32-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-31-0ec1c3ff-7154-4fd1-a090-8fa0b9c2d3c8-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-4-8b99da05-6be1-42e1-90e0-bff8ca08b5af-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-14-5b39012d-85e3-4144-ac83-574ed5046718-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-15-e37e3f98-5f3b-4245-9c01-76cf0f1dd21a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-19-e36a1955-f01c-4e08-9f19-dcec0cbceadd-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-20-c9632a29-fd5b-4ba5-a0db-2992c73075c2-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-28-31220e43-f85a-4609-9ebb-4c76269cddb6-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-3-7e47d440-44f5-4a11-bbfa-c72c520ca997-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-7-c902540c-b002-413c-b422-7ec3c06d404e-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-11-5558f0e5-7a46-417e-8c0a-50bb00e54a30-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-12-ea94b7c7-164d-4d06-8b4d-24d296906f73-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-22-0565b9eb-060d-4180-aa1e-2bb69187d8db-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-25-1f2b05c6-5407-41fc-a881-60626253ca77-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/af797bab-2f2c-44df-a77b-d91c7198fe53-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-8270633197658268308-1-af797bab-2f2c-44df-a77b-d91c7198fe53.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,436 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/27
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 27
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 15: Code-Review+1
(8 comments)
Thank you very much for your quick and great work! Only found a few nits, other than that LGTM!
http://gerrit.cloudera.org:8080/#/c/16143/15/fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
File fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/15/fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java@66
PS15, Line 66: iceberg table not supported truncate.
nit: "we also cannot truncate Iceberg tables."
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README
File testdata/data/README:
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@503
PS15, Line 503: Icebreg
nit: Iceberg
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@507
PS15, Line 507: including table metadata
nit: "which contains table metadata"
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@507
PS15, Line 507: managerd
managed
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/data/README@508
PS15, Line 508: including data files
nit: which contains the data files.
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test:
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test@19
PS15, Line 19: ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:
: CAUSED BY: IllegalArgumentException: Cannot find source column: event_time
Can we catch this error during analysis, and throw an AnalysisException instead of ImpalaRuntimeException?
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test:
http://gerrit.cloudera.org:8080/#/c/16143/15/testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test@7
PS15, Line 7: row_regex: .* PARQUET/GZIP:4
: row_regex: .* PARQUET/GZIP:2
I'm just wondering how deterministic is it. Do the test consistently pass if you run it multiple times?
The alternative is to use
SUM(aggregation, RowsRead)
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test:
http://gerrit.cloudera.org:8080/#/c/16143/13/testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test@77
PS13, Line 77:
> 1. I've already add non-count queries in iceberg-query.test
The generated test files usually contain blank lines which needs to be removed. You can remove the blank lines in vim with the following:
:g/^$/d
Other than that it's up to you if you just copy over the original iceberg-query.test file, or just copy-paste some parts of it. But before committing the file don't forget to check that you only modified/added tests that you intended.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 15
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 20 Jul 2020 17:44:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6537/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 3
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Jul 2020 03:11:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
IMPALA-9741: Supported query Icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't identity this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When query iceberg table, we pushdown
partition column predicates to iceberg to decided which data files
need to be scanned, and then transformed these information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/custom_cluster/test_iceberg.py
82 files changed, 1,304 insertions(+), 159 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/5
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 5
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6351/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 27 Aug 2020 23:13:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 27:
> FileMetadataLoaderTest.testIcebergLoading and test_create_iceberg_tables
> still fail:
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11899/testReport/
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/
Hi Zoltan, sorry about this. I forgot to modify data file name in FileMetadataLoaderTest.java. I've already adjust code and restart a Jenkins test.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 27
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 14:37:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#25). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-1-5dbd44ad-18bc-40f2-9dd6-aeb2cc23457c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-3-27db2521-1e8b-40c1-b846-552cd620abce-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-4-f1b55628-0544-4833-8b11-1b4add53dfd6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-6-f75530ef-93b6-4994-b3c8-db957d44848c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-7-8d9b22da-5f10-4cbf-8e4d-160f829b5e48-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-9-f029a1f7-9024-4bc3-a030-e20861586146-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-11-f07814ae-56cd-486b-af81-18541437da7d-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-12-967c70a4-bf4d-4a82-8c97-c90e2b4d9dcf-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-14-d0cdca7f-c050-407e-b70c-2bd076f83e4e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-15-0e931a1f-309e-43b3-a5cf-3ef82fa4a87c-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-17-43138078-244c-4b38-8127-04a5bfbc4695-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-19-52569895-df25-4ad8-b64d-49c4540d36c9-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-20-f160c1ea-a2f5-4109-81ec-3ff9c155430f-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-22-c1f61b8c-9d9a-4823-b64e-109770c16696-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-23-8cfc9d65-bfc3-47c4-8da7-6610b49b3305-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-25-d519b09d-c730-40a6-ad03-8b36096cb234-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-27-802b2129-0d18-418a-bcdd-b1b5332ea2a5-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-28-70f51a0c-3d30-4cdb-afcb-e8c1cfa3caa4-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-30-e887707d-58db-469c-ab96-b77188f25189-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-31-facf6a62-c326-44f7-bdc3-b1471bd594d7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-93996984692289973-1-9b8c72ab-43b9-42fb-a5e9-1dcfa1801a21.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-1-b975a171-0911-47c2-90c8-300f23c28772-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-6-9bc6f547-de52-4aad-8c07-5d0f1aaa0558-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-9-bf3024d6-8b55-42d3-b25e-5f2a31303026-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-17-a0d23269-c8d7-40cf-94c8-8b624969b9c0-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-23-138a376e-696e-4da3-95f3-7e0aacb11ca3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-27-04494175-75de-41ef-b824-3923ebfba0cb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-30-eba34050-b263-41e8-bc7d-773a47b33c32-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-31-0ec1c3ff-7154-4fd1-a090-8fa0b9c2d3c8-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-4-8b99da05-6be1-42e1-90e0-bff8ca08b5af-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-14-5b39012d-85e3-4144-ac83-574ed5046718-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-15-e37e3f98-5f3b-4245-9c01-76cf0f1dd21a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-19-e36a1955-f01c-4e08-9f19-dcec0cbceadd-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-20-c9632a29-fd5b-4ba5-a0db-2992c73075c2-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-28-31220e43-f85a-4609-9ebb-4c76269cddb6-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-3-7e47d440-44f5-4a11-bbfa-c72c520ca997-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-7-c902540c-b002-413c-b422-7ec3c06d404e-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-11-5558f0e5-7a46-417e-8c0a-50bb00e54a30-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-12-ea94b7c7-164d-4d06-8b4d-24d296906f73-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-22-0565b9eb-060d-4180-aa1e-2bb69187d8db-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-25-1f2b05c6-5407-41fc-a881-60626253ca77-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/af797bab-2f2c-44df-a77b-d91c7198fe53-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-8270633197658268308-1-af797bab-2f2c-44df-a77b-d91c7198fe53.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
89 files changed, 1,436 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/25
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 21:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7032/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 21
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 13:44:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6354/
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 14:20:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20:
(9 comments)
> (10 comments)
>
> One thing I didn't understand about the patch is the support for
> partition pruning. I don't see enough code here to handle all the
> edge cases - partition evolution, different partition transforms,
> etc.
>
> I'm OK with deferring optimizations to a later patch, but want to
> make sure that this patch is correct and that we document what the
> limitations are.
Hi Tim, thanks for your patient review. This patch pruning partition mainly by pushdown predicates to Iceberg to filter data files. I've already add more comment about predicates pushdown. And all test cases passed: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11658/
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@611
PS18, Line 611: // Iceberg table only has one partition spec now
> What does this mean? We don't support partition evolution yet?
When we create iceberg table with several partition columns, such as 'id identity', 'register_time day', these partition columns are treated as partition fields, all in a partition spec which is different from hdfs table. So I add a comment here: Iceberg table only has one partition.
If we alter iceberg table's partitions, iceberg will generated snapshot with new partition spec, we can select different snapshot to query by iceberg api which is not supported in this patch. We only get the latest partition spec for latest data now, so we think iceberg table only has one partition spec in impala(latest partition spec)
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@615
PS18, Line 615: fieldName
> nit: fieldName
Done
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1889
PS18, Line 1889: protec
> Can you make this protected instead of public, since it's only being used w
Done
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@372
PS18, Line 372: protec
> I think this can be protected
Done
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@61
PS18, Line 61:
> nit: icebergConjuncts_
Done
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@94
PS18, Line 94:
> nit: Iceberg
Done
http://gerrit.cloudera.org:8080/#/c/16143/18/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@123
PS18, Line 123: ListIterator<Expr> it = conjuncts_.listIterator();
> I think this predicate pushdown needs a bit more explanation.
Yes you are right, Tim. In case 1 and case 2, we need to evaluate predicates in the scan as well. After discussed with my colleague worked on Iceberg, we prepare to pushdown all predicates to Iceberg, and evaluate all predicates in the scan as well. More details you can see the comment on IcebergScanNode.extractIcebergConjuncts
http://gerrit.cloudera.org:8080/#/c/16143/18/testdata/data/README
File testdata/data/README:
http://gerrit.cloudera.org:8080/#/c/16143/18/testdata/data/README@508
PS18, Line 508: hll_sketches_from_impala.parquet:
> nit: remove this merge marker
Done
http://gerrit.cloudera.org:8080/#/c/16143/18/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test:
http://gerrit.cloudera.org:8080/#/c/16143/18/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test@3
PS18, Line 3: Iceberg
> nit: Iceberg
Done
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 02:13:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Supported query icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query icebreg table by impala
......................................................................
IMPALA-9741: Supported query icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently support
PARQUET and ORC. And if you don't identity this property in your SQL,
default file format is PARQUT.
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
23 files changed, 852 insertions(+), 135 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/2
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 2
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 22:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7049/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 22
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 06:44:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 31:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6405/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 31
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sat, 05 Sep 2020 20:54:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 25:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7052/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 11:30:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 21:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java@86
PS20, Line 86: partitioned = true;
> nit: probably just call this variable 'partitioned', having a negation in a
Done
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@134
PS20, Line 134: // LocalFsTable transformed from iceberg table only has one partition
> nit: then maybe we could use a Preconditions check to verify this. Also, we
Done
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@59
PS20, Line 59: icebergConjuncts
> icebergConjuncts_
Done
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@60
PS20, Line 60: icebergPredicates
> icebergPredicates_
Done
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@117
PS20, Line 117: 1
> In case 1 we cannot filter data files because the column is not part of the
We can filter data files even column is not part of the partition expressions in some situation. Iceberg will reserve some data files' statistics in it's metadata, so we can use these non-partition column predicates to filter data files, just like impala use parquet file statistics to skip unnecessary file scan.
http://gerrit.cloudera.org:8080/#/c/16143/20/testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
File testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json:
PS20:
> So there's a problem with these test tables. They have hdfs://localhost:205
Hi Tim, do you mean that I need to use '$DEFAULT_FS' to replace 'hdfs://localhost:20500' in these metadata files, and then replace '$DEFAULT_FS' by real file system before data loading? If so, would you please give me some advice, such as how to get real file system, which script to be modified is better and so on. Thanks a lot!
http://gerrit.cloudera.org:8080/#/c/16143/20/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test:
http://gerrit.cloudera.org:8080/#/c/16143/20/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test@4
PS20, Line 4: SELECT count(*) from ic
> nit: it can be omitted
Done
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 21
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 13:31:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#24). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
45 files changed, 1,436 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/24
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 24
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 26:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7056/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 26
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 07:13:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 28:
> (1 comment)
>
> No worries, thanks for your persistence. Seems like something is
> wrong with test_iceberg.py in the dockerised test. I'm also trying
> to figure out what went wrong.
>
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/query_test.test_iceberg/TestCreatingIcebergTable/test_create_iceberg_tables_protocol__beeswax___exec_option____batch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0____table_format__parquet_none_/
I add file_format constrint in test_iceberg.py. But when I test single case by: impala-py.test tests/query_test/test_iceberg.py. Whenever I add file_format constrint, all passed, so I'm not sure if this modify worked.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 03:26:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 31: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 31
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sun, 06 Sep 2020 02:12:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6653/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 14
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 20 Jul 2020 11:55:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6613/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 8
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 16 Jul 2020 06:54:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 28:
I did a dockerised build to get some insights why test_create_iceberg_tables fails in dockerised tests.
The test fails at "DESCRIBE iceberg_test_external_empty_column;"
To get more insight I switched to "DESCRIBE formatted". You can check the output at https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3091/testReport/query_test.test_iceberg/TestCreatingIcebergTable/test_create_iceberg_tables_protocol__beeswax___exec_option____batch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0____table_format__parquet_none_/
I don't know yet why it couldn't figure out the table schema.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 12:47:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 32:
Well done, and thank you for your awesome work!
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 32
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sun, 06 Sep 2020 19:48:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6654/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 15
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 20 Jul 2020 11:58:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
IMPALA-9741: Supported query Icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't identity this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When query iceberg table, we pushdown
partition column predicates to iceberg to decided which data files
need to be scanned, and then transformed these information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/bin/generate-schema-statements.py
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/functional-query_core.csv
M testdata/workloads/functional-query/functional-query_dimensions.csv
M testdata/workloads/functional-query/functional-query_exhaustive.csv
M testdata/workloads/functional-query/functional-query_pairwise.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/common/test_dimensions.py
M tests/custom_cluster/test_iceberg.py
88 files changed, 1,359 insertions(+), 163 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/6
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 6
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6630/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 11
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 17 Jul 2020 02:35:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support quering Icebreg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support quering Icebreg table by impala
......................................................................
Patch Set 11:
(9 comments)
Nice work! Only found some nit comments/grammar mistakes.
Now I'm waiting to see some end-to-end tests.
Please don't forget to add tests about showing table metadata, e.g., SHOW PARTITIONS, SHOW FILES, etc.
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@7
PS11, Line 7: quering Icebreg
misspelled querying
Also, still misspelled Iceberg
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@9
PS11, Line 9: query
nit: querying
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@30
PS11, Line 30: query
querying
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@31
PS11, Line 31: decided
decide
http://gerrit.cloudera.org:8080/#/c/16143/11//COMMIT_MSG@32
PS11, Line 32: transformed these
transfer this
http://gerrit.cloudera.org:8080/#/c/16143/11/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/16143/11/common/thrift/CatalogObjects.thrift@516
PS11, Line 516: s
nit: Plural 's' not needed since the value type of the map is not a list
http://gerrit.cloudera.org:8080/#/c/16143/11/common/thrift/CatalogObjects.thrift@516
PS11, Line 516: 5: optional map<string,THdfsFileDesc> path_md5_to_file_descriptors
nit: please add comment about field 5.
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/6/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java@88
PS6, Line 88: if (table_ instanceof FeIcebergTable) {
: if (((FeIcebergTable) table_).getSourceIdToPartitionMap().isEmpty()) {
: notPartitioned = true;
: }
> Yes, you are right, we treated iceberg table as unpartitioned hdfs table, b
I see, thanks for the explanation.
http://gerrit.cloudera.org:8080/#/c/16143/11/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/11/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@258
PS11, Line 258: PartitionColToSourceId
SourceIdToPartition
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 11
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 17 Jul 2020 12:07:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 19:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/16143/19/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/19/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java@89
PS19, Line 89: // We only get latest partition spec from Iceberg now, so this list only contains
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/19/fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java@93
PS19, Line 93: // Iceberg table with partition spec without partition fields is non-partitioned.
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/19/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/19/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@115
PS19, Line 115: * 2.The column is part of all partition keys without any transformation (i.e. IDENTITY);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16143/19/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@117
PS19, Line 117: * We can use case 1 and 3 to filter data files, but also need to evaluate it in the scan,
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 19
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 12 Aug 2020 08:14:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 20:
(3 comments)
I am close to a +1/+2 once you've addressed Zoltan's comments.
I think there is a problem with the localhost hardcoded in the .json and .avro test data files though that will prevent data being loaded on many configurations. We could fix it if we replaced hdfs://localhost:20500/ with $DEFAULT_FS, but I'm not sure how best to do that with avro.
I'm going to run a precommit dry run to confirm that the dockerised tests will fail.
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@59
PS20, Line 59: icebergConjunts_
icebergConjuncts_
http://gerrit.cloudera.org:8080/#/c/16143/20/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@60
PS20, Line 60: icebergPredicate_
icebergPredicates_
http://gerrit.cloudera.org:8080/#/c/16143/20/testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
File testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json:
PS20:
So there's a problem with these test tables. They have hdfs://localhost:20500 hardcoded in the json and also in the avro files.
This needs to be substituted with $DEFAULT_FS somehow for it to work on alternative filesystems. We support running tests on non-hdfs filesystems and with alternative namenode hostnames.
This patch won't pass the precommit tests because the dockerized build use a different default fs
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 27 Aug 2020 23:13:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 18:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6821/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 18
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 07 Aug 2020 06:17:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 28:
(2 comments)
I was able to figure what was the problem with test_create_iceberg_tables.
You can reproduce the error with local catalog:
bin/start-impala-cluster.py --impalad_args --enable_minidumps=false --impalad_args --use_local_catalog=true --catalogd_args --catalog_topic_mode=minimal
bin/impala-py.test tests/query_test/test_iceberg.py::TestCreatingIcebergTable::test_create_iceberg_tables
The problem is that in CatalogService.doGetPartialCatalogObject() we use 'IcebergTable.getHdfsTable()' in the response.
IcebergTable.hdfsTable_ is not necessarily the latest version of the table, as it is not updated in IcebergTable.load().
It means in the test we just return the very first, columnless version of the table.
I've added comments at the relevant places.
In my environment there's another problem unfortunately, as one of the DROP TABLE statements at the end of the test fails.
http://gerrit.cloudera.org:8080/#/c/16143/28/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/16143/28/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3199
PS28, Line 3199: ((IcebergTable) table).getHdfsTable();
Here we return IcebergTable.hdfsTable_, which might be obsolete. See IcebergTable.load().
http://gerrit.cloudera.org:8080/#/c/16143/28/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/16143/28/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@185
PS28, Line 185: hdfsTable_.load(false, msClient, msTbl, true, true, false, null, reason);
We load 'hdfsTable_' here, however the schema might be updated at L192. Because of this CatalogServiceCatalog's doGetPartialCatalogObject() will return this old version.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 03 Sep 2020 14:26:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
Patch Set 17:
(4 comments)
Hi anjalinorwood, thanks for your review!
http://gerrit.cloudera.org:8080/#/c/16143/16//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16143/16//COMMIT_MSG@29
PS16, Line 29: We achieved this function by treating the iceberg table as normal
> Are there plans to support read of Iceberg table as a partitioned table? Th
Yes, you are right, partitioned table is useful for query plan. But we may not consider this in our first version, since it's hard to treat Iceberg table as an partitioned hdfs table.
But we will definitely do this in next version, including: compute incremental stats/query plan optimization for iceberg and so on. This patch is just a simple version to scan iceberg table.
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java@80
PS16, Line 80: }
> The double negative is pretty hard to parse.
Done
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@139
PS16, Line 139: if (!(predicate.getChild(0) instanceof SlotRef)) return false;
> Can the predicate be of the form: '10 = p1'? In that case, should there be
yes, it is. I've already test by debug, sql like this: select * from table where 0=id can also pushdown predicate to iceberg.
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@114
PS16, Line 114: if ("PARQUET".equalsIgnoreCase(format)) return TIcebergFileFormat.PARQUET;
> Rest of the code seems to support Iceberg ORC file format. This code does n
I supported ORC format in original version, but when I test scan iceberg table with ORC, I found exception: https://issues.apache.org/jira/browse/IMPALA-9967
So I removed this, maybe supported more file format in other patch.
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 17
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 12:40:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Supported query Icebreg table by impala
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
IMPALA-9741: Supported query Icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't identity this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When query iceberg table, we pushdown
partition column predicates to iceberg to decided which data files
need to be scanned, and then transformed these information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/custom_cluster/test_iceberg.py
82 files changed, 1,304 insertions(+), 159 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/7
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 7
Gerrit-Owner: wangsheng <sk...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>