You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2020/12/07 17:15:01 UTC
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16825
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
23 files changed, 400 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/1
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7825/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 10 Dec 2020 12:08:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16825/8/be/src/exec/hdfs-table-sink.cc
File be/src/exec/hdfs-table-sink.cc:
http://gerrit.cloudera.org:8080/#/c/16825/8/be/src/exec/hdfs-table-sink.cc@458
PS8, Line 458: table_desc_->col_descs()[j].name()
Columns[j] doesn't necessarily correspond to partition_key_expr_evals_[j] for Iceberg tables.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Sat, 12 Dec 2020 18:07:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7843/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 11 Dec 2020 15:01:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 14: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 17 Dec 2020 03:16:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16825
to look at the new patch set (#5).
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
TODO:
* Current change includes some parts of IMPALA-10384 which needs to
be removed once https://gerrit.cloudera.org/#/c/16850/ is merged
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
25 files changed, 438 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/5
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16825
to look at the new patch set (#10).
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
TODO:
* Current change includes some parts of IMPALA-10384 which needs to
be removed once https://gerrit.cloudera.org/#/c/16850/ is merged
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
25 files changed, 532 insertions(+), 61 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/10
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7867/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 16 Dec 2020 17:13:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7792/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 08 Dec 2020 17:14:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6736/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 08 Dec 2020 16:53:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 7:
(3 comments)
Thanks for new feature, Zoltan. Just some nits.
http://gerrit.cloudera.org:8080/#/c/16825/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16825/4//COMMIT_MSG@7
PS4, Line 7: identity-partitioned Iceberg tables
Shall we use "Iceberg tables with only 'IDENTITY' partition" here?
http://gerrit.cloudera.org:8080/#/c/16825/4/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/16825/4/common/thrift/CatalogService.thrift@194
PS4, Line 194: tional string tab
Maybe 'TIcebergOperationParam' here? These variable is not operation itself, just related to operation.
http://gerrit.cloudera.org:8080/#/c/16825/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/16825/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4655
PS4, Line 4655: ToCreate.isEmpty()) {
It there any difference between 'partName' and partName.substring(0, partName.length())?
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 10 Dec 2020 12:41:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 7:
PS7 is a rebase that resolves the merge conflict.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 10 Dec 2020 11:57:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16825
to look at the new patch set (#8).
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
TODO:
* Current change includes some parts of IMPALA-10384 which needs to
be removed once https://gerrit.cloudera.org/#/c/16850/ is merged
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
25 files changed, 439 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/8
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16825/3/be/src/exec/hdfs-table-sink.cc
File be/src/exec/hdfs-table-sink.cc:
http://gerrit.cloudera.org:8080/#/c/16825/3/be/src/exec/hdfs-table-sink.cc@264
PS3, Line 264:
Fix indentation
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 08 Dec 2020 17:45:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 14: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 17 Dec 2020 08:54:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 9:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/7860/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 15 Dec 2020 13:22:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6732/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 07 Dec 2020 18:09:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16825
to look at the new patch set (#7).
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
TODO:
* Current change includes some parts of IMPALA-10384 which needs to
be removed once https://gerrit.cloudera.org/#/c/16850/ is merged
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
25 files changed, 438 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/7
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16825
to look at the new patch set (#4).
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
TODO:
* Current change includes IMPALA-10384, maybe we should push it
separately
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
24 files changed, 405 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/4
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 11: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6781/
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 15 Dec 2020 23:40:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6757/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 10 Dec 2020 11:57:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 13: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 16 Dec 2020 20:39:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 11:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6781/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 15 Dec 2020 18:03:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6743/
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 09 Dec 2020 23:36:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16825
to look at the new patch set (#13).
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
The INSERT INTO syntax is similar to the syntax for non-partitioned
tables, i.e.:
INSERT INTO <iceberg_tbl> VALUES (<val1>, <val2>, <val3>, ...);
Or,
INSERT INTO <iceberg_tbl> SELECT <val1>, <val2>, ... FROM <source_tbl>
(please note that we don't use the PARTITION keyword)
The values must be in column order corresponding to the table schema.
Impala will automatically create/find the partitions based on the
Iceberg partition spec.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
27 files changed, 588 insertions(+), 91 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/13
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 12:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/16825/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16825/11//COMMIT_MSG@35
PS11, Line 35: TODO:
: * Current change includes some parts of IMPALA-10384 which needs to
: be removed once https://gerrit.cloudera.org/#/c/16850/ is merged
https://gerrit.cloudera.org/#/c/16850/ is already merged
http://gerrit.cloudera.org:8080/#/c/16825/12//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16825/12//COMMIT_MSG@12
PS12, Line 12: Identity-partitioned Iceberg tables are similar to regular
: partitioned tables
Can you mention the difference in INSERT syntax? I mean that there is no PARTITION(x=y ... ) in the statement, instead select list -> partition mapping occurs bases on column order if I understand correctly.
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/exec/hdfs-table-sink.cc
File be/src/exec/hdfs-table-sink.cc:
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/exec/hdfs-table-sink.cc@157
PS11, Line 157: // For Iceberg tables we only have a single partition descriptor.
: if (IsIceberg()) break;
This is the same as skipping the whole for loop, right? I would prefer to put the for loop in an if block, or if we want to avoid additional nesting, the extract the loop to a separate function.
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/exec/hdfs-table-sink.cc@579
PS11, Line 579: it != partition_descriptor_map_.end()
It it possible for this to succeed if IsIceberg() is true? If not, then I think it is clearer to handle the IsIceberg() case first.
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/runtime/descriptors.cc
File be/src/runtime/descriptors.cc:
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/runtime/descriptors.cc@250
PS11, Line 250: TIcebergPartitionSpec
Maybe const ref would be better to avoid copy.
http://gerrit.cloudera.org:8080/#/c/16825/11/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test:
http://gerrit.cloudera.org:8080/#/c/16825/11/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test@4
PS11, Line 4: partitioend
typo
http://gerrit.cloudera.org:8080/#/c/16825/12/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test:
http://gerrit.cloudera.org:8080/#/c/16825/12/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test@128
PS12, Line 128: select id, bool_col, int_col, bigint_col, float_col, double_col,
Can you add test here or in iceberg-negative.test where the insert is rejected because the types in the select list do not match the types in the column? E.g the columns here could be reordered.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 16 Dec 2020 15:26:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7826/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 10 Dec 2020 12:16:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 12: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 16 Dec 2020 18:38:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6732/
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 07 Dec 2020 19:29:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6757/
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 10 Dec 2020 21:57:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7786/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Mon, 07 Dec 2020 17:36:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7861/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 15 Dec 2020 18:19:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6743/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 09 Dec 2020 13:36:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7864/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 16 Dec 2020 13:23:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 9:
PS9 was a rebase to resolve merge conflicts.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Tue, 15 Dec 2020 13:14:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7806/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 09 Dec 2020 13:55:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16825
to look at the new patch set (#3).
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
TODO:
* Current change includes IMPALA-10384, maybe we should push it
separately
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
24 files changed, 404 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/3
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16825
to look at the new patch set (#9).
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
TODO:
* Current change includes some parts of IMPALA-10384 which needs to
be removed once https://gerrit.cloudera.org/#/c/16850/ is merged
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
24 files changed, 431 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/9
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 12:
(7 comments)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/16825/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16825/11//COMMIT_MSG@35
PS11, Line 35: TODO:
: * Current change includes some parts of IMPALA-10384 which needs to
: be removed once https://gerrit.cloudera.org/#/c/16850/ is merged
> https://gerrit.cloudera.org/#/c/16850/ is already merged
Done
http://gerrit.cloudera.org:8080/#/c/16825/12//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16825/12//COMMIT_MSG@12
PS12, Line 12: Identity-partitioned Iceberg tables are similar to regular
: partitioned tables
> Can you mention the difference in INSERT syntax? I mean that there is no PA
Done
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/exec/hdfs-table-sink.cc
File be/src/exec/hdfs-table-sink.cc:
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/exec/hdfs-table-sink.cc@157
PS11, Line 157: // For Iceberg tables we only have a single partition descriptor.
: if (IsIceberg()) break;
> This is the same as skipping the whole for loop, right? I would prefer to p
Extracted the loop to a separate function.
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/exec/hdfs-table-sink.cc@579
PS11, Line 579: it != partition_descriptor_map_.end()
> It it possible for this to succeed if IsIceberg() is true? If not, then I t
Done
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/runtime/descriptors.cc
File be/src/runtime/descriptors.cc:
http://gerrit.cloudera.org:8080/#/c/16825/11/be/src/runtime/descriptors.cc@250
PS11, Line 250: TIcebergPartitionSpec
> Maybe const ref would be better to avoid copy.
Thanks for catching this. Some programming in Java and Python and I surely forget these peculiarities.
http://gerrit.cloudera.org:8080/#/c/16825/11/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test:
http://gerrit.cloudera.org:8080/#/c/16825/11/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test@4
PS11, Line 4: partitioend
> typo
Done
http://gerrit.cloudera.org:8080/#/c/16825/12/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test:
http://gerrit.cloudera.org:8080/#/c/16825/12/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test@128
PS12, Line 128: select id, bool_col, int_col, bigint_col, float_col, double_col,
> Can you add test here or in iceberg-negative.test where the insert is rejec
Done
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 16 Dec 2020 16:59:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
The INSERT INTO syntax is similar to the syntax for non-partitioned
tables, i.e.:
INSERT INTO <iceberg_tbl> VALUES (<val1>, <val2>, <val3>, ...);
Or,
INSERT INTO <iceberg_tbl> SELECT <val1>, <val2>, ... FROM <source_tbl>
(please note that we don't use the PARTITION keyword)
The values must be in column order corresponding to the table schema.
Impala will automatically create/find the partitions based on the
Iceberg partition spec.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Reviewed-on: http://gerrit.cloudera.org:8080/16825
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
27 files changed, 588 insertions(+), 91 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 15
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16825
to look at the new patch set (#12).
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
This patch adds support to INSERT INTO identity-partitioned
Iceberg tables.
Identity-partitioned Iceberg tables are similar to regular
partitioned tables, they are even stored in the same directory
structure. The difference is that the data files still store
the partitioning columns.
Partitioned Iceberg tables are stored as non-partitioned tables
in the Hive Metastore (similarly to partitioned Kudu tables). However,
the InsertStmt still generates the partition expressions for them.
These partition expressions are used to shuffle and sort the input
data so we don't end up writing too many files. The HdfsTableSink
also uses the partition expressions to write the data files with
the proper partition paths.
Iceberg is able to parse the partition paths to generate the
corresponding metadata for the partitions. This happens at the
end in IcebergCatalogOpExecutor.
Testing:
* added planner test to verify shuffling and sorting
* added negative tests for unsupported features like PARTITION clause
and non-identity partition transforms
* e2e tests with partitioned inserts
TODO:
* Current change includes some parts of IMPALA-10384 which needs to
be removed once https://gerrit.cloudera.org/#/c/16850/ is merged
Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/dml-exec-state.cc
M be/src/service/client-request-state.cc
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
M tests/query_test/test_iceberg.py
27 files changed, 558 insertions(+), 83 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16825/12
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6736/
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 09 Dec 2020 02:53:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "wangsheng (Code Review)" <ge...@cloudera.org>.
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 13: Code-Review+2
Thanks for modify the code
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 17 Dec 2020 03:15:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6790/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Thu, 17 Dec 2020 03:16:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6785/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 16 Dec 2020 13:02:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO identity-partitioned Iceberg tables
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16825/3/be/src/exec/hdfs-table-sink.cc
File be/src/exec/hdfs-table-sink.cc:
http://gerrit.cloudera.org:8080/#/c/16825/3/be/src/exec/hdfs-table-sink.cc@264
PS3, Line 264: S
> Fix indentation
Done
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Wed, 09 Dec 2020 13:36:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16825 )
Change subject: IMPALA-10380: INSERT INTO Iceberg tables with 'IDENTITY' partitions only
......................................................................
Patch Set 8:
(3 comments)
Thanks for the comments, WangSheng!
http://gerrit.cloudera.org:8080/#/c/16825/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16825/4//COMMIT_MSG@7
PS4, Line 7: Iceberg tables with 'IDENTITY' part
> Shall we use "Iceberg tables with only 'IDENTITY' partition" here?
Updated the title
http://gerrit.cloudera.org:8080/#/c/16825/4/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/16825/4/common/thrift/CatalogService.thrift@194
PS4, Line 194: tional string tab
> Maybe 'TIcebergOperationParam' here? These variable is not operation itself
Done
http://gerrit.cloudera.org:8080/#/c/16825/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/16825/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4655
PS4, Line 4655: ToCreate.isEmpty()) {
> It there any difference between 'partName' and partName.substring(0, partNa
I think the only difference is that it creates an unnecessary copy of the String object. Thanks for spotting this, I also fixed it in my CR for IMPALA-10384: https://gerrit.cloudera.org/#/c/16850/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
I'll remove these partition name related changes once that fix is merged.
--
To view, visit http://gerrit.cloudera.org:8080/16825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If98797a2bfdc038d0467c8f83aadf1a12e1d69d4
Gerrit-Change-Number: 16825
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: wangsheng <sk...@163.com>
Gerrit-Comment-Date: Fri, 11 Dec 2020 14:44:24 +0000
Gerrit-HasComments: Yes