You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Attila Jeges (Code Review)" <ge...@cloudera.org> on 2021/07/23 16:38:38 UTC
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Attila Jeges has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17723
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
IMPALA-10739: Support setting new partition spec for Iceberg tables
With this patch Impala will support partition evolution for
Iceberg tables.
The DDL statement to change the default partition spec is:
ALTER TABLE <tbl> SET PARTITION SPEC(<partition-spec>)
Hive uses rhe same SQL syntax.
Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
---
M be/src/exec/hdfs-table-sink.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
A fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
10 files changed, 295 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/17723/1
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 1
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9219/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 2
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Aug 2021 12:20:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
IMPALA-10739: Support setting new partition spec for Iceberg tables
With this patch Impala will support partition evolution for
Iceberg tables.
The DDL statement to change the default partition spec is:
ALTER TABLE <tbl> SET PARTITION SPEC(<partition-spec>)
Hive uses the same SQL syntax.
Testing:
- Added FE test to exercise parsing various well-formed and ill-formed
ALTER TABLE SET PARTITION SPEC statements.
- Added e2e tests for:
- ALTER TABLE SET PARTITION SPEC works for tables with HadoopTables
and HadoopCatalog Catalog.
- When evolving partition spec, the old data written with an earlier
spec remains unchanged. New data is written using the new spec in
a new layout. Data written with earlier spec and new spec can be
fetched in a single query.
- Invalid ALTER TABLE SET PARTITION SPEC statements yield the
expected analysis error messages.
Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Reviewed-on: http://gerrit.cloudera.org:8080/17723
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-table-sink.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
A fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
10 files changed, 306 insertions(+), 3 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 5
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9157/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 1
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 23 Jul 2021 16:59:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 3: Code-Review+2
Great job!
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 3
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 09:40:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7363/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 4
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 10:20:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Attila Jeges (Code Review)" <ge...@cloudera.org>.
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/17723/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17723/1//COMMIT_MSG@15
PS1, Line 15: rhe
> the
Done
http://gerrit.cloudera.org:8080/#/c/17723/1//COMMIT_MSG@16
PS1, Line 16:
> Please add section about testing.
Done
http://gerrit.cloudera.org:8080/#/c/17723/1/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/17723/1/common/thrift/JniCatalog.thrift@469
PS1, Line 469: from
> for
Done
http://gerrit.cloudera.org:8080/#/c/17723/1/fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java:
http://gerrit.cloudera.org:8080/#/c/17723/1/fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java@50
PS1, Line 50: sb.append(getTbl()).append(" SET PARTITION SPEC ")
> Do we need to add parenthesis here, or are they added by icebergPartSpec_.t
The parentheses are added in icebergPartSpec_.toSql()
http://gerrit.cloudera.org:8080/#/c/17723/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test:
http://gerrit.cloudera.org:8080/#/c/17723/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test@303
PS1, Line 303: ---- TYPES
> Could you please add a SHOW FILES statement at the end so we can see the di
Done
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 1
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Aug 2021 11:57:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Attila Jeges (Code Review)" <ge...@cloudera.org>.
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17723/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test:
http://gerrit.cloudera.org:8080/#/c/17723/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test@309
PS2, Line 309: .*.0.parq','.*',''
> This matches to all data files. Can we exclude '=' from the first .*?
Good catch! Done.
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 2
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 08:55:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 1: Code-Review+1
(5 comments)
The change looks really good. I only had some minor comments.
http://gerrit.cloudera.org:8080/#/c/17723/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17723/1//COMMIT_MSG@15
PS1, Line 15: rhe
the
http://gerrit.cloudera.org:8080/#/c/17723/1//COMMIT_MSG@16
PS1, Line 16:
Please add section about testing.
http://gerrit.cloudera.org:8080/#/c/17723/1/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/17723/1/common/thrift/JniCatalog.thrift@469
PS1, Line 469: from
for
http://gerrit.cloudera.org:8080/#/c/17723/1/fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java:
http://gerrit.cloudera.org:8080/#/c/17723/1/fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java@50
PS1, Line 50: sb.append(getTbl()).append(" SET PARTITION SPEC ")
Do we need to add parenthesis here, or are they added by icebergPartSpec_.toSql()?
http://gerrit.cloudera.org:8080/#/c/17723/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test:
http://gerrit.cloudera.org:8080/#/c/17723/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test@303
PS1, Line 303: ---- TYPES
Could you please add a SHOW FILES statement at the end so we can see the different partitions?
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 1
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jul 2021 13:24:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 4
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 16:27:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Attila Jeges (Code Review)" <ge...@cloudera.org>.
Attila Jeges has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
IMPALA-10739: Support setting new partition spec for Iceberg tables
With this patch Impala will support partition evolution for
Iceberg tables.
The DDL statement to change the default partition spec is:
ALTER TABLE <tbl> SET PARTITION SPEC(<partition-spec>)
Hive uses the same SQL syntax.
Testing:
- Added FE test to exercise parsing various well-formed and ill-formed
ALTER TABLE SET PARTITION SPEC statements.
- Added e2e tests for:
- ALTER TABLE SET PARTITION SPEC works for tables with HadoopTables
and HadoopCatalog Catalog.
- When evolving partition spec, the old data written with an earlier
spec remains unchanged. New data is written using the new spec in
a new layout. Data written with earlier spec and new spec can be
fetched in a single query.
- Invalid ALTER TABLE SET PARTITION SPEC statements yield the
expected analysis error messages.
Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
---
M be/src/exec/hdfs-table-sink.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
A fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
10 files changed, 306 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/17723/3
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 3
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17723/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test:
http://gerrit.cloudera.org:8080/#/c/17723/2/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test@309
PS2, Line 309: .*.0.parq','.*',''
This matches to all data files. Can we exclude '=' from the first .*?
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 2
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Aug 2021 18:10:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Attila Jeges (Code Review)" <ge...@cloudera.org>.
Attila Jeges has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
IMPALA-10739: Support setting new partition spec for Iceberg tables
With this patch Impala will support partition evolution for
Iceberg tables.
The DDL statement to change the default partition spec is:
ALTER TABLE <tbl> SET PARTITION SPEC(<partition-spec>)
Hive uses the same SQL syntax.
Testing:
- Added FE test to exercise parsing various well-formed and ill-formed
ALTER TABLE SET PARTITION SPEC statements.
- Added e2e tests for:
- ALTER TABLE SET PARTITION SPEC works for tables with HadoopTables
and HadoopCatalog Catalog.
- When evolving partition spec, the old data written with an earlier
spec remains unchanged. New data is written using the new spec in
a new layout. Data written with earlier spec and new spec can be
fetched in a single query.
- Invalid ALTER TABLE SET PARTITION SPEC statements yield the
expected analysis error messages.
Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
---
M be/src/exec/hdfs-table-sink.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
A fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
10 files changed, 306 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/17723/2
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 2
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 4
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 10:20:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10739: Support setting new partition spec for Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17723 )
Change subject: IMPALA-10739: Support setting new partition spec for Iceberg tables
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9224/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bd935b8a82e977df9ee90d464b5fe2a7acc83f2
Gerrit-Change-Number: 17723
Gerrit-PatchSet: 3
Gerrit-Owner: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Attila Jeges <at...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 09:18:38 +0000
Gerrit-HasComments: No