You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Anonymous Coward (Code Review)" <ge...@cloudera.org> on 2023/01/12 11:31:45 UTC
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
lipenglin@apache.org has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19416
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Table property 'external.table.purge' should not be ignored when
creating Iceberg tables, except that when 'iceberg.catalog' is not the
Hive Catalog for managed tables, because we need to call
'org.apache.hadoop.hive.metastore.IMetaStoreClient#createTable' and HMS
will override 'external.table.purge' to 'TRUE'.
Testing:
* existing tests
* add e2e tests
Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
6 files changed, 90 insertions(+), 25 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/19416/1
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <li...@apache.org>
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@768
PS1, Line 768: // HMS will override 'external.table.purge' to 'TRUE' When 'iceberg.catalog' is not
: // the Hive Catalog for managed tables.
> Sure! I should create a Jira in the Hive community, right?
Yeah, I just tested the following in Hive:
create table test_parq_hive (i int)
stored as parquet
tblproperties ('external.table.purge'='false');
Then checked the table properties via the DESCRIBE FORMATTED command:
...
external.table.purge | TRUE
...
I think it's worth to raise a Jira for this.
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Jan 2023 14:37:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 2:
(1 comment)
https://issues.apache.org/jira/browse/HIVE-26959?focusedCommentId=17678103&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17678103
It seems that we can only warn users that the property 'external.table.purge' will be ignored when creating an Iceberg table if the catalog is not the Hive Catalog, because without external keyword Hive will consider like we are trying for managed table, which will set 'external.table.purge' to TRUE.
At least, if we warn the user and they won't be confused by explicitly set 'external.table.purge' to FALSE but it is really TRUE.
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191
PS1, Line 191: TRUE
> What is the reason behind setting the default to true for external tables?
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ExternalTables
Since Hive 4.0.0, this is set to true by default, but if we don't want to drop the data when dropping the table, we should set it to false when creating the table or with the 'alter table' statement.
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Jan 2023 07:03:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 2: Code-Review+2
(1 comment)
Thank you for the clarifications! LGTM!
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191
PS1, Line 191: TRUE
> There are two scenarios(TL;DR):
Thank you for taking time explaining. I see your point, I think you were talking about managed table's perspective while I was focusing on external tables.
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Feb 2023 10:43:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@apache.org has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Table property 'external.table.purge' should not be ignored when
creating Iceberg tables, except that when 'iceberg.catalog' is not the
Hive Catalog for managed tables, because we need to call
'org.apache.hadoop.hive.metastore.IMetaStoreClient#createTable' and HMS
will override 'external.table.purge' to 'TRUE'.
Testing:
* existing tests
* add e2e tests
Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
6 files changed, 90 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/19416/2
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 2:
(1 comment)
Thanks for comments!
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191
PS1, Line 191: TRUE
> Thank you for the explanation, I was not aware that the entire external.tab
There are two scenarios(TL;DR):
<-- Dividing line -->
1 execute 'create table' statement
Query1: create table ice_tbl (col_1 int) stored as iceberg;
Query2:
show create table ice_tbl
"CREATE EXTERNAL TABLE xyz.ice_tbl (
col_1 INT NULL
)
STORED AS ICEBERG
LOCATION 'hdfs://localhost:20500/test-warehouse/xyz.db/ice_tbl'
TBLPROPERTIES ('external.table.purge'='TRUE', ...)"
'external.table.purge' will be true by default, because without 'external' keyword Hive will consider like we are trying for managed table, so Hive set it to true by default for CREATE TABLE statement without 'external' keyword.
<-- Dividing line -->
2 execute 'create external table' statement
Query1:
create external table ice_tbl_ext
stored as iceberg
location '/data/ice_tbl_ext'
tblproperties('iceberg.catalog' = 'hadoop.tables');
Query2:
show create table ice_v2_update_all_rows
"CREATE EXTERNAL TABLE xyz.ice_tbl_ext (
col_int INT NOT NULL,
col_str STRING NOT NULL
)
STORED AS ICEBERG
LOCATION 'hdfs://localhost:20500/data/ice_tbl_ext'
TBLPROPERTIES (...)"
'external.table.purge' will be empty, but if we delete this table 'ice_tbl_ext', the data under 'hdfs://localhost:20500/data/ice_tbl_ext' will not be deleted, because "hive.external.table.purge.default" is false https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L3660.
<-- Dividing line -->
So we set TBL_PROP_EXTERNAL_TABLE_PURGE_DEFAULT -> TRUE is consistent with Hive, because it only applies to CREATE TABLE statement without 'external' keyword.
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Feb 2023 11:38:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Feb 2023 11:48:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
LGTM, thanks for fixing this.
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@768
PS1, Line 768: // HMS will override 'external.table.purge' to 'TRUE' When 'iceberg.catalog' is not
: // the Hive Catalog for managed tables.
Do you want to open a Jira for this?
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 18:03:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Table property 'external.table.purge' should not be ignored when
creating Iceberg tables, except that when 'iceberg.catalog' is not the
Hive Catalog for managed tables, because we need to call
'org.apache.hadoop.hive.metastore.IMetaStoreClient#createTable' and HMS
will override 'external.table.purge' to 'TRUE'.
Testing:
* existing tests
* add e2e tests
Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Reviewed-on: http://gerrit.cloudera.org:8080/19416
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
6 files changed, 90 insertions(+), 23 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12251/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Jan 2023 06:57:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@768
PS1, Line 768: // HMS will override 'external.table.purge' to 'TRUE' When 'iceberg.catalog' is not
: // the Hive Catalog for managed tables.
> Yeah, I just tested the following in Hive:
I created a jira to track this:
https://issues.apache.org/jira/browse/HIVE-26959
And Impala cannot guarantee that the user is using a fixed version of HMS, so this warn message maybe should be worth keeping here.
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Jan 2023 05:58:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@768
PS1, Line 768: // HMS will override 'external.table.purge' to 'TRUE' When 'iceberg.catalog' is not
: // the Hive Catalog for managed tables.
> Do you want to open a Jira for this?
Sure! I should create a Jira in the Hive community, right?
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Jan 2023 10:26:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9020/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Feb 2023 11:48:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Feb 2023 17:00:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191
PS1, Line 191: TRUE
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#Languag
Thank you for the explanation, I was not aware that the entire external.table.purge property was introduced to handle migrated tables.
Hive seems to set it to false by default, should we follow this direction? It sounds more natural to me to keep it false by default and set it to true for edge cases like migrated table or Iceberg HiveCatalog tables. Or we could move this property to IcebergTable.
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L3660
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Feb 2023 09:43:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12153/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Jan 2023 11:51:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19416 )
Change subject: IMPALA-11798: Property 'external.table.purge' should not be ignored when CREATE Iceberg tables
......................................................................
Patch Set 1:
(1 comment)
Thanks for this nice change! I only had one question about the default value.
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/19416/1/fe/src/main/java/org/apache/impala/catalog/Table.java@191
PS1, Line 191: TRUE
What is the reason behind setting the default to true for external tables?
--
To view, visit http://gerrit.cloudera.org:8080/19416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2649dd38fbe050044817d6c425ef447245aa2829
Gerrit-Change-Number: 19416
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Jan 2023 10:06:59 +0000
Gerrit-HasComments: Yes