You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by bo...@apache.org on 2024/03/20 13:30:57 UTC
(impala) branch master updated (c02bac01c -> 814acf3bb)
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from c02bac01c IMPALA-12924: Upgrade xmlsec to 2.2.6
new eb5c8d688 IMPALA-12802: Support ALTER TABLE for JDBC tables
new 814acf3bb IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../impala/analysis/AlterTableAddColsStmt.java | 10 ++
.../AlterTableAddDropRangePartitionStmt.java | 3 +
.../analysis/AlterTableAddPartitionStmt.java | 8 ++
.../impala/analysis/AlterTableAlterColStmt.java | 12 ++
.../impala/analysis/AlterTableDropColStmt.java | 3 +
.../analysis/AlterTableDropPartitionStmt.java | 8 ++
.../AlterTableExecuteExpireSnapshotsStmt.java | 3 +
.../analysis/AlterTableExecuteRollbackStmt.java | 3 +
.../impala/analysis/AlterTableExecuteStmt.java | 3 +
.../analysis/AlterTableOrViewRenameStmt.java | 3 +
.../analysis/AlterTableOrViewSetOwnerStmt.java | 3 +
.../analysis/AlterTableRecoverPartitionsStmt.java | 3 +
.../impala/analysis/AlterTableReplaceColsStmt.java | 3 +
.../impala/analysis/AlterTableSetCachedStmt.java | 5 +
.../impala/analysis/AlterTableSetColumnStats.java | 3 +
.../analysis/AlterTableSetFileFormatStmt.java | 3 +
.../impala/analysis/AlterTableSetLocationStmt.java | 3 +
.../impala/analysis/AlterTableSetOwnerStmt.java | 3 +
.../analysis/AlterTableSetPartitionSpecStmt.java | 3 +
.../analysis/AlterTableSetRowFormatStmt.java | 3 +
.../apache/impala/analysis/AlterTableSetStmt.java | 4 +-
.../analysis/AlterTableSetTblProperties.java | 44 ++++++-
.../impala/analysis/AlterTableSortByStmt.java | 3 +
.../org/apache/impala/analysis/AlterTableStmt.java | 17 ++-
.../analysis/AlterTableUnSetTblProperties.java | 32 +++++
.../impala/analysis/AlterViewSetOwnerStmt.java | 3 +
.../impala/analysis/AlterViewSetTblProperties.java | 3 +
.../analysis/AlterViewUnSetTblProperties.java | 3 +
.../org/apache/impala/catalog/DataSourceTable.java | 18 +++
.../apache/impala/catalog/FeDataSourceTable.java | 1 +
.../impala/catalog/local/LocalDataSourceTable.java | 6 +
.../org/apache/impala/analysis/AnalyzeDDLTest.java | 145 +++++++++++++++++----
...eberg-alter.test => iceberg-alter-default.test} | 2 +-
.../{iceberg-alter.test => iceberg-alter-v1.test} | 99 ++------------
.../{iceberg-alter.test => iceberg-alter-v2.test} | 26 ++--
...est => iceberg-partitioned-insert-default.test} | 0
...ert.test => iceberg-partitioned-insert-v1.test} | 27 ++--
...ert.test => iceberg-partitioned-insert-v2.test} | 45 ++++---
.../queries/QueryTest/jdbc-data-source.test | 120 +++++++++++++++++
tests/query_test/test_iceberg.py | 51 +++++---
40 files changed, 557 insertions(+), 180 deletions(-)
copy testdata/workloads/functional-query/queries/QueryTest/{iceberg-alter.test => iceberg-alter-default.test} (99%)
copy testdata/workloads/functional-query/queries/QueryTest/{iceberg-alter.test => iceberg-alter-v1.test} (77%)
rename testdata/workloads/functional-query/queries/QueryTest/{iceberg-alter.test => iceberg-alter-v2.test} (96%)
copy testdata/workloads/functional-query/queries/QueryTest/{iceberg-partitioned-insert.test => iceberg-partitioned-insert-default.test} (100%)
copy testdata/workloads/functional-query/queries/QueryTest/{iceberg-partitioned-insert.test => iceberg-partitioned-insert-v1.test} (98%)
rename testdata/workloads/functional-query/queries/QueryTest/{iceberg-partitioned-insert.test => iceberg-partitioned-insert-v2.test} (96%)
(impala) 02/02: IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 814acf3bbf015ca305603d33dada8f4233d929d0
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Mon Mar 11 15:02:30 2024 +0100
IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests
This CR is the first step to upgrade to Iceberg 1.4.3. The biggest
change in behavior in Iceberg 1.4.3 is that Iceberg V2 tables are
the default. Because of this we update some test files to
explicitly create V1/V2 tables. We also introduce test files that
create Iceberg tables without explicitly specifying the format
version, these tests have the name *-default.test. The latter tests
will need to be updated when we actually upgrade to Iceberg 1.4.3.
Change-Id: Ieb4f6c1b206d1d4fd878f07ea5f1436dcae560cd
Reviewed-on: http://gerrit.cloudera.org:8080/21167
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Andrew Sherman <as...@cloudera.com>
---
...eberg-alter.test => iceberg-alter-default.test} | 2 +-
.../{iceberg-alter.test => iceberg-alter-v1.test} | 99 ++--------------------
.../{iceberg-alter.test => iceberg-alter-v2.test} | 26 +++---
...est => iceberg-partitioned-insert-default.test} | 0
...ert.test => iceberg-partitioned-insert-v1.test} | 27 ++++--
...ert.test => iceberg-partitioned-insert-v2.test} | 45 ++++++----
tests/query_test/test_iceberg.py | 51 +++++++----
7 files changed, 105 insertions(+), 145 deletions(-)
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-default.test
similarity index 99%
copy from testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
copy to testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-default.test
index 5d20f8660..c1ef2185e 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-default.test
@@ -492,4 +492,4 @@ DESCRIBE ice_alter_cols;
'd','decimal(22,3)','','true'
---- TYPES
STRING,STRING,STRING,STRING
-====
\ No newline at end of file
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-v1.test
similarity index 77%
copy from testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
copy to testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-v1.test
index 5d20f8660..2310f669f 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-v1.test
@@ -4,7 +4,7 @@ CREATE TABLE iceberg_hadoop_tables(
level STRING
)
STORED AS ICEBERG
-TBLPROPERTIES('iceberg.catalog'='hadoop.tables');
+TBLPROPERTIES('format-version'='1', 'iceberg.catalog'='hadoop.tables');
ALTER TABLE iceberg_hadoop_tables ADD COLUMNS(event_time TIMESTAMP, register_time DATE);
ALTER TABLE iceberg_hadoop_tables ADD COLUMNS(message STRING, price DECIMAL(8,1));
ALTER TABLE iceberg_hadoop_tables ADD COLUMNS(map_test MAP <STRING, array <STRING>>, struct_test STRUCT <f1: BIGINT, f2: BIGINT>);
@@ -73,7 +73,7 @@ CREATE TABLE iceberg_hadoop_catalog(
level STRING
)
STORED AS ICEBERG
-TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+TBLPROPERTIES('format-version'='1', 'iceberg.catalog'='hadoop.catalog',
'iceberg.catalog_location'='$WAREHOUSE_LOCATION_PREFIX/other/$DATABASE/hadoop_catalog_test');
ALTER TABLE iceberg_hadoop_catalog ADD COLUMNS(event_time TIMESTAMP, register_time DATE);
ALTER TABLE iceberg_hadoop_catalog ADD COLUMNS(message STRING, price DECIMAL(8,1));
@@ -132,7 +132,7 @@ STRING, BIGINT, BIGINT
---- QUERY
CREATE TABLE iceberg_rename (i int)
STORED AS ICEBERG
-TBLPROPERTIES('iceberg.catalog'='hive.catalog');
+TBLPROPERTIES('format-version'='1', 'iceberg.catalog'='hive.catalog');
INSERT INTO iceberg_rename values (42);
ALTER TABLE iceberg_rename RENAME TO iceberg_rename2;
SELECT * FROM iceberg_rename2;
@@ -149,7 +149,7 @@ Could not resolve table reference: 'iceberg_rename'
---- QUERY
CREATE TABLE iceberg_changing_fileformats (i int)
STORED AS ICEBERG
-TBLPROPERTIES('write.format.default'='orc');
+TBLPROPERTIES('format-version'='1', 'write.format.default'='orc');
DESCRIBE FORMATTED iceberg_changing_fileformats;
---- RESULTS: VERIFY_IS_SUBSET
'','write.format.default','orc '
@@ -186,7 +186,8 @@ string, string, string
====
---- QUERY
CREATE TABLE ice_alter_cols (f float)
-STORED AS ICEBERG;
+STORED AS ICEBERG
+TBLPROPERTIES('format-version'='1');
INSERT INTO ice_alter_cols VALUES (2.718);
SELECT * FROM ice_alter_cols;
---- RESULTS
@@ -315,6 +316,7 @@ STRING,STRING,STRING,STRING
CREATE TABLE iceberg_changing_parq_tblprops (i int)
STORED AS ICEBERG
TBLPROPERTIES (
+ 'format-version'='1',
'write.parquet.row-group-size-bytes'='134217728',
'write.parquet.compression-codec'='zstd',
'write.parquet.compression-level'='12',
@@ -352,91 +354,6 @@ DESCRIBE FORMATTED iceberg_changing_parq_tblprops;
string, string, string
====
---- QUERY
-CREATE TABLE iceberg_upgrade_v2_no_write_mode (i INT) STORED AS ICEBERG;
-DESCRIBE FORMATTED iceberg_upgrade_v2_no_write_mode;
----- RESULTS: VERIFY_IS_NOT_IN
-'','write.delete.mode ','merge-on-read '
-'','write.update.mode ','merge-on-read '
-'','write.merge.mode ','merge-on-read '
----- TYPES
-string, string, string
-====
----- QUERY
-# Setting format-version to 1 doesn't add write modes.
-ALTER TABLE iceberg_upgrade_v2_no_write_mode SET TBLPROPERTIES('format-version'='1');
-DESCRIBE FORMATTED iceberg_upgrade_v2_no_write_mode;
----- RESULTS: VERIFY_IS_NOT_IN
-'','write.delete.mode ','merge-on-read '
-'','write.update.mode ','merge-on-read '
-'','write.merge.mode ','merge-on-read '
----- TYPES
-string, string, string
-====
----- QUERY
-ALTER TABLE iceberg_upgrade_v2_no_write_mode SET TBLPROPERTIES('format-version'='2');
-DESCRIBE FORMATTED iceberg_upgrade_v2_no_write_mode;
----- RESULTS: VERIFY_IS_SUBSET
-'','write.delete.mode ','merge-on-read '
-'','write.update.mode ','merge-on-read '
-'','write.merge.mode ','merge-on-read '
----- TYPES
-string, string, string
-====
----- QUERY
-CREATE TABLE iceberg_upgrade_v2_delete_mode (i INT) STORED AS ICEBERG;
-ALTER TABLE iceberg_upgrade_v2_delete_mode
-SET TBLPROPERTIES('format-version'='2', 'write.delete.mode'='copy-on-write');
-DESCRIBE FORMATTED iceberg_upgrade_v2_delete_mode;
----- RESULTS: VERIFY_IS_SUBSET
-'','write.delete.mode ','copy-on-write '
----- TYPES
-string, string, string
-====
----- QUERY
-DESCRIBE FORMATTED iceberg_upgrade_v2_delete_mode;
----- RESULTS: VERIFY_IS_NOT_IN
-'','write.update.mode ','merge-on-read '
-'','write.merge.mode ','merge-on-read '
----- TYPES
-string, string, string
-====
----- QUERY
-CREATE TABLE iceberg_upgrade_v2_update_mode (i INT) STORED AS ICEBERG;
-ALTER TABLE iceberg_upgrade_v2_update_mode
-SET TBLPROPERTIES('format-version'='2', 'write.update.mode'='copy-on-write');
-DESCRIBE FORMATTED iceberg_upgrade_v2_update_mode;
----- RESULTS: VERIFY_IS_SUBSET
-'','write.update.mode ','copy-on-write '
----- TYPES
-string, string, string
-====
----- QUERY
-DESCRIBE FORMATTED iceberg_upgrade_v2_update_mode;
----- RESULTS: VERIFY_IS_NOT_IN
-'','write.delete.mode ','merge-on-read '
-'','write.merge.mode ','merge-on-read '
----- TYPES
-string, string, string
-====
----- QUERY
-CREATE TABLE iceberg_upgrade_v2_merge_mode (i INT) STORED AS ICEBERG;
-ALTER TABLE iceberg_upgrade_v2_merge_mode
-SET TBLPROPERTIES('format-version'='2', 'write.merge.mode'='merge-on-read');
-DESCRIBE FORMATTED iceberg_upgrade_v2_merge_mode;
----- RESULTS: VERIFY_IS_SUBSET
-'','write.merge.mode ','merge-on-read '
----- TYPES
-string, string, string
-====
----- QUERY
-DESCRIBE FORMATTED iceberg_upgrade_v2_merge_mode;
----- RESULTS: VERIFY_IS_NOT_IN
-'','write.update.mode ','merge-on-read '
-'','write.delete.mode ','merge-on-read '
----- TYPES
-string, string, string
-====
----- QUERY
# Add a column that already exists and a new column that does not exist with
# "if not exists" clause.
ALTER TABLE ice_alter_cols ADD IF NOT EXISTS COLUMNS (a bigint, d bigint)
@@ -492,4 +409,4 @@ DESCRIBE ice_alter_cols;
'd','decimal(22,3)','','true'
---- TYPES
STRING,STRING,STRING,STRING
-====
\ No newline at end of file
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-v2.test
similarity index 96%
rename from testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
rename to testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-v2.test
index 5d20f8660..e99c4a5ae 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-v2.test
@@ -4,7 +4,7 @@ CREATE TABLE iceberg_hadoop_tables(
level STRING
)
STORED AS ICEBERG
-TBLPROPERTIES('iceberg.catalog'='hadoop.tables');
+TBLPROPERTIES('format-version'='2', 'iceberg.catalog'='hadoop.tables');
ALTER TABLE iceberg_hadoop_tables ADD COLUMNS(event_time TIMESTAMP, register_time DATE);
ALTER TABLE iceberg_hadoop_tables ADD COLUMNS(message STRING, price DECIMAL(8,1));
ALTER TABLE iceberg_hadoop_tables ADD COLUMNS(map_test MAP <STRING, array <STRING>>, struct_test STRUCT <f1: BIGINT, f2: BIGINT>);
@@ -73,7 +73,7 @@ CREATE TABLE iceberg_hadoop_catalog(
level STRING
)
STORED AS ICEBERG
-TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+TBLPROPERTIES('format-version'='2', 'iceberg.catalog'='hadoop.catalog',
'iceberg.catalog_location'='$WAREHOUSE_LOCATION_PREFIX/other/$DATABASE/hadoop_catalog_test');
ALTER TABLE iceberg_hadoop_catalog ADD COLUMNS(event_time TIMESTAMP, register_time DATE);
ALTER TABLE iceberg_hadoop_catalog ADD COLUMNS(message STRING, price DECIMAL(8,1));
@@ -132,7 +132,7 @@ STRING, BIGINT, BIGINT
---- QUERY
CREATE TABLE iceberg_rename (i int)
STORED AS ICEBERG
-TBLPROPERTIES('iceberg.catalog'='hive.catalog');
+TBLPROPERTIES('format-version'='2', 'iceberg.catalog'='hive.catalog');
INSERT INTO iceberg_rename values (42);
ALTER TABLE iceberg_rename RENAME TO iceberg_rename2;
SELECT * FROM iceberg_rename2;
@@ -149,7 +149,7 @@ Could not resolve table reference: 'iceberg_rename'
---- QUERY
CREATE TABLE iceberg_changing_fileformats (i int)
STORED AS ICEBERG
-TBLPROPERTIES('write.format.default'='orc');
+TBLPROPERTIES('format-version'='2', 'write.format.default'='orc');
DESCRIBE FORMATTED iceberg_changing_fileformats;
---- RESULTS: VERIFY_IS_SUBSET
'','write.format.default','orc '
@@ -186,7 +186,8 @@ string, string, string
====
---- QUERY
CREATE TABLE ice_alter_cols (f float)
-STORED AS ICEBERG;
+STORED AS ICEBERG
+TBLPROPERTIES('format-version'='2');
INSERT INTO ice_alter_cols VALUES (2.718);
SELECT * FROM ice_alter_cols;
---- RESULTS
@@ -315,6 +316,7 @@ STRING,STRING,STRING,STRING
CREATE TABLE iceberg_changing_parq_tblprops (i int)
STORED AS ICEBERG
TBLPROPERTIES (
+ 'format-version'='2',
'write.parquet.row-group-size-bytes'='134217728',
'write.parquet.compression-codec'='zstd',
'write.parquet.compression-level'='12',
@@ -352,7 +354,8 @@ DESCRIBE FORMATTED iceberg_changing_parq_tblprops;
string, string, string
====
---- QUERY
-CREATE TABLE iceberg_upgrade_v2_no_write_mode (i INT) STORED AS ICEBERG;
+CREATE TABLE iceberg_upgrade_v2_no_write_mode (i INT) STORED AS ICEBERG
+TBLPROPERTIES ('format-version'='1');
DESCRIBE FORMATTED iceberg_upgrade_v2_no_write_mode;
---- RESULTS: VERIFY_IS_NOT_IN
'','write.delete.mode ','merge-on-read '
@@ -383,7 +386,8 @@ DESCRIBE FORMATTED iceberg_upgrade_v2_no_write_mode;
string, string, string
====
---- QUERY
-CREATE TABLE iceberg_upgrade_v2_delete_mode (i INT) STORED AS ICEBERG;
+CREATE TABLE iceberg_upgrade_v2_delete_mode (i INT) STORED AS ICEBERG
+TBLPROPERTIES('format-version'='1');
ALTER TABLE iceberg_upgrade_v2_delete_mode
SET TBLPROPERTIES('format-version'='2', 'write.delete.mode'='copy-on-write');
DESCRIBE FORMATTED iceberg_upgrade_v2_delete_mode;
@@ -401,7 +405,8 @@ DESCRIBE FORMATTED iceberg_upgrade_v2_delete_mode;
string, string, string
====
---- QUERY
-CREATE TABLE iceberg_upgrade_v2_update_mode (i INT) STORED AS ICEBERG;
+CREATE TABLE iceberg_upgrade_v2_update_mode (i INT) STORED AS ICEBERG
+TBLPROPERTIES('format-version'='1');
ALTER TABLE iceberg_upgrade_v2_update_mode
SET TBLPROPERTIES('format-version'='2', 'write.update.mode'='copy-on-write');
DESCRIBE FORMATTED iceberg_upgrade_v2_update_mode;
@@ -419,7 +424,8 @@ DESCRIBE FORMATTED iceberg_upgrade_v2_update_mode;
string, string, string
====
---- QUERY
-CREATE TABLE iceberg_upgrade_v2_merge_mode (i INT) STORED AS ICEBERG;
+CREATE TABLE iceberg_upgrade_v2_merge_mode (i INT) STORED AS ICEBERG
+TBLPROPERTIES('format-version'='1');
ALTER TABLE iceberg_upgrade_v2_merge_mode
SET TBLPROPERTIES('format-version'='2', 'write.merge.mode'='merge-on-read');
DESCRIBE FORMATTED iceberg_upgrade_v2_merge_mode;
@@ -492,4 +498,4 @@ DESCRIBE ice_alter_cols;
'd','decimal(22,3)','','true'
---- TYPES
STRING,STRING,STRING,STRING
-====
\ No newline at end of file
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-default.test
similarity index 100%
copy from testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
copy to testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-default.test
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-v1.test
similarity index 98%
copy from testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
copy to testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-v1.test
index da70926fa..6c35c4818 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-v1.test
@@ -4,7 +4,8 @@
# the partitioned column.
create table ice_only_part (i int)
partitioned by spec (i)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='1');
---- RESULTS
'Table has been created.'
====
@@ -61,7 +62,8 @@ aggregation(SUM, RowsRead): 1
# Test inserts with multple partition columns.
create table ice_multi_part (i int, d date, s string)
partitioned by spec(i, d)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='1');
---- RESULTS
'Table has been created.'
====
@@ -157,7 +159,8 @@ aggregation(SUM, RowsRead): 3
# Test that Impala only writes one file per partitions.
create table ice_bigints (i BIGINT, j BIGINT, k BIGINT)
partitioned by spec (i, j)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='1');
---- RESULTS
'Table has been created.'
====
@@ -216,7 +219,8 @@ partitioned by spec (
double_col,
date_col,
string_col)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='1');
---- RESULTS
'Table has been created.'
====
@@ -403,7 +407,8 @@ STRING, STRING, STRING, STRING
# Test inserts with multple partition columns.
create table ice_part_non_order (i int, d date, s string)
partitioned by spec(s, d)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='1');
---- RESULTS
'Table has been created.'
====
@@ -439,7 +444,8 @@ STRING, BIGINT, BIGINT
---- QUERY
# Test inserts with different partition specs.
create table ice_alter_part (i int, d date, s string)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='1');
---- RESULTS
'Table has been created.'
====
@@ -516,7 +522,8 @@ STRING, BIGINT, BIGINT
---- QUERY
create table ice_void (i int, s string, d date)
partitioned by spec (void(i), truncate(1, s), year(d))
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='1');
insert into ice_void values (1, 'one', '2001-01-01'),
(11,'other','2001-01-11'),
(2, 'two', '2002-02-02'),
@@ -668,7 +675,8 @@ SHOW PARTITIONS ice_void;
STRING, BIGINT, BIGINT
====
---- QUERY
-create table store_sales partitioned by spec (ss_sold_date_sk) stored as iceberg
+create table store_sales partitioned by spec (ss_sold_date_sk)
+stored as iceberg tblproperties ('format-version'='1')
as select * from tpcds_parquet.store_sales;
select count(*) from store_sales;
---- RESULTS
@@ -688,7 +696,8 @@ BIGINT
# Check that result strings are not URL encoded.
create table special_char_partitions (i int, s string, s2 string)
partitioned by spec (i, s, truncate(4, s))
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='1');
insert into special_char_partitions
values (1, '11/14/31', '44/1'), (2, '11"14"31', '43"3'), (3, '11=14=31', '65=2'), (4, '', 'a'), (5, cast(null as string), 'b');
select * from special_char_partitions;
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-v2.test
similarity index 96%
rename from testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
rename to testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-v2.test
index da70926fa..c65326d17 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-v2.test
@@ -4,7 +4,8 @@
# the partitioned column.
create table ice_only_part (i int)
partitioned by spec (i)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='2');
---- RESULTS
'Table has been created.'
====
@@ -61,7 +62,8 @@ aggregation(SUM, RowsRead): 1
# Test inserts with multple partition columns.
create table ice_multi_part (i int, d date, s string)
partitioned by spec(i, d)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='2');
---- RESULTS
'Table has been created.'
====
@@ -157,7 +159,8 @@ aggregation(SUM, RowsRead): 3
# Test that Impala only writes one file per partitions.
create table ice_bigints (i BIGINT, j BIGINT, k BIGINT)
partitioned by spec (i, j)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='2');
---- RESULTS
'Table has been created.'
====
@@ -216,7 +219,8 @@ partitioned by spec (
double_col,
date_col,
string_col)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='2');
---- RESULTS
'Table has been created.'
====
@@ -403,7 +407,8 @@ STRING, STRING, STRING, STRING
# Test inserts with multple partition columns.
create table ice_part_non_order (i int, d date, s string)
partitioned by spec(s, d)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='2');
---- RESULTS
'Table has been created.'
====
@@ -439,7 +444,8 @@ STRING, BIGINT, BIGINT
---- QUERY
# Test inserts with different partition specs.
create table ice_alter_part (i int, d date, s string)
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='2');
---- RESULTS
'Table has been created.'
====
@@ -507,8 +513,8 @@ SHOW PARTITIONS ice_alter_part;
---- RESULTS
'{"i":"3","d":"18605"}',1,1
'{"i":"4","d":"18606"}',1,1
-'{"i":"5","d":null,"d_year":"50","s_bucket_5":"2"}',1,1
-'{"i":"6","d":null,"d_year":"50","s_bucket_5":"0"}',1,1
+'{"i":"5","d_year":"50","s_bucket_5":"2"}',1,1
+'{"i":"6","d_year":"50","s_bucket_5":"0"}',1,1
'{}',2,1
---- TYPES
STRING, BIGINT, BIGINT
@@ -516,7 +522,8 @@ STRING, BIGINT, BIGINT
---- QUERY
create table ice_void (i int, s string, d date)
partitioned by spec (void(i), truncate(1, s), year(d))
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='2');
insert into ice_void values (1, 'one', '2001-01-01'),
(11,'other','2001-01-11'),
(2, 'two', '2002-02-02'),
@@ -578,10 +585,10 @@ STRING, STRING, STRING, STRING
---- QUERY
SHOW PARTITIONS ice_void;
---- RESULTS
+'{"d_year":"34","i":"4","s_null":null}',1,1
'{"i_null":null,"s_trunc":"o","d_year":"31"}',2,1
'{"i_null":null,"s_trunc":"t","d_year":"32"}',1,1
'{"i_null":null,"s_trunc":"t","d_year":"33"}',1,1
-'{"i_null":null,"s_trunc":null,"d_year":"34","i":"4","s_null":null}',1,1
---- TYPES
STRING, BIGINT, BIGINT
====
@@ -616,11 +623,11 @@ STRING, STRING, STRING, STRING
---- QUERY
SHOW PARTITIONS ice_void;
---- RESULTS
+'{"d_year":"34","i":"4","s_null":null}',1,1
+'{"i":"5","s_null":null,"d_null":null}',2,1
'{"i_null":null,"s_trunc":"o","d_year":"31"}',2,1
'{"i_null":null,"s_trunc":"t","d_year":"32"}',1,1
'{"i_null":null,"s_trunc":"t","d_year":"33"}',1,1
-'{"i_null":null,"s_trunc":null,"d_year":"34","i":"4","s_null":null}',1,1
-'{"i_null":null,"s_trunc":null,"d_year":null,"i":"5","s_null":null,"d_null":null}',2,1
---- TYPES
STRING, BIGINT, BIGINT
====
@@ -658,17 +665,18 @@ STRING, STRING, STRING, STRING
---- QUERY
SHOW PARTITIONS ice_void;
---- RESULTS
+'{"d_year":"34","i":"4","s_null":null}',1,1
+'{"i":"5","s_null":null,"d_null":null}',2,1
'{"i_null":null,"s_trunc":"o","d_year":"31"}',2,1
'{"i_null":null,"s_trunc":"t","d_year":"32"}',1,1
'{"i_null":null,"s_trunc":"t","d_year":"33"}',1,1
-'{"i_null":null,"s_trunc":null,"d_year":"34","i":"4","s_null":null}',1,1
-'{"i_null":null,"s_trunc":null,"d_year":null,"i":"5","s_null":null,"d_null":null}',2,1
-'{"i_null":null,"s_trunc":null,"d_year":null,"i":null,"s_null":null,"d_null":null}',2,1
+'{"s_null":null,"d_null":null,"i_null":null}',2,1
---- TYPES
STRING, BIGINT, BIGINT
====
---- QUERY
-create table store_sales partitioned by spec (ss_sold_date_sk) stored as iceberg
+create table store_sales partitioned by spec (ss_sold_date_sk)
+stored as iceberg tblproperties ('format-version'='2')
as select * from tpcds_parquet.store_sales;
select count(*) from store_sales;
---- RESULTS
@@ -688,7 +696,8 @@ BIGINT
# Check that result strings are not URL encoded.
create table special_char_partitions (i int, s string, s2 string)
partitioned by spec (i, s, truncate(4, s))
-stored as iceberg;
+stored as iceberg
+tblproperties ('format-version'='2');
insert into special_char_partitions
values (1, '11/14/31', '44/1'), (2, '11"14"31', '43"3'), (3, '11=14=31', '65=2'), (4, '', 'a'), (5, cast(null as string), 'b');
select * from special_char_partitions;
@@ -790,7 +799,7 @@ show partitions special_char_partitions;
'{"i":"3","s":"11=14=31","s_trunc":"11=1"}',1,1
'{"i":"4","s":"","s_trunc":""}',1,1
'{"i":"5","s":null,"s_trunc":null}',1,1
-'{"i":null,"s":null,"s_trunc":null,"s2":"98\\/22"}',1,1
+'{"s2":"98\\/22"}',1,1
---- TYPES
STRING,BIGINT,BIGINT
====
diff --git a/tests/query_test/test_iceberg.py b/tests/query_test/test_iceberg.py
index a6cca7ffc..ce57ae725 100644
--- a/tests/query_test/test_iceberg.py
+++ b/tests/query_test/test_iceberg.py
@@ -69,8 +69,14 @@ class TestIcebergTable(IcebergTestSuite):
def test_create_iceberg_tables(self, vector, unique_database):
self.run_test_case('QueryTest/iceberg-create', vector, use_db=unique_database)
- def test_alter_iceberg_tables(self, vector, unique_database):
- self.run_test_case('QueryTest/iceberg-alter', vector, use_db=unique_database)
+ def test_alter_iceberg_tables_v1(self, vector, unique_database):
+ self.run_test_case('QueryTest/iceberg-alter-v1', vector, use_db=unique_database)
+
+ def test_alter_iceberg_tables_v2(self, vector, unique_database):
+ self.run_test_case('QueryTest/iceberg-alter-v2', vector, use_db=unique_database)
+
+ def test_alter_iceberg_tables_default(self, vector, unique_database):
+ self.run_test_case('QueryTest/iceberg-alter-default', vector, use_db=unique_database)
def test_external_iceberg_tables(self, vector, unique_database):
self.run_test_case('QueryTest/iceberg-external', vector, unique_database)
@@ -179,8 +185,16 @@ class TestIcebergTable(IcebergTestSuite):
def test_insert(self, vector, unique_database):
self.run_test_case('QueryTest/iceberg-insert', vector, use_db=unique_database)
- def test_partitioned_insert(self, vector, unique_database):
- self.run_test_case('QueryTest/iceberg-partitioned-insert', vector,
+ def test_partitioned_insert_v1(self, vector, unique_database):
+ self.run_test_case('QueryTest/iceberg-partitioned-insert-v1', vector,
+ use_db=unique_database)
+
+ def test_partitioned_insert_v2(self, vector, unique_database):
+ self.run_test_case('QueryTest/iceberg-partitioned-insert-v2', vector,
+ use_db=unique_database)
+
+ def test_partitioned_insert_default(self, vector, unique_database):
+ self.run_test_case('QueryTest/iceberg-partitioned-insert-default', vector,
use_db=unique_database)
def test_insert_overwrite(self, vector, unique_database):
@@ -783,10 +797,10 @@ class TestIcebergTable(IcebergTestSuite):
# Create table
table_name = "ice_part"
qualified_table_name = "%s.%s" % (unique_database, table_name)
- create_table = 'create table %s ' \
- '(s string, i int) partitioned by spec(truncate(5, s), identity(i)) ' \
- 'stored as iceberg' \
- % qualified_table_name
+ create_table = """create table {}
+ (s string, i int) partitioned by spec(truncate(5, s), identity(i))
+ stored as iceberg
+ tblproperties ('format-version'='1')""".format(qualified_table_name)
self.client.execute(create_table)
partition_spec = self.get_current_partition_spec(unique_database, table_name)
@@ -878,17 +892,22 @@ class TestIcebergTable(IcebergTestSuite):
assert truncate_s['field-id'] == 1004
@SkipIf.not_dfs
- def test_writing_metrics_to_metadata(self, vector, unique_database):
+ def test_writing_metrics_to_metadata_v1(self, vector, unique_database):
+ self._test_writing_metrics_to_metadata_impl(unique_database, 'ice_stats_v1', '1')
+
+ @SkipIf.not_dfs
+ def test_writing_metrics_to_metadata_v2(self, vector, unique_database):
+ self._test_writing_metrics_to_metadata_impl(unique_database, 'ice_stats_v2', '2')
+
+ def _test_writing_metrics_to_metadata_impl(self, unique_database, table_name, version):
# Create table
- table_name = "ice_stats"
qualified_table_name = "%s.%s" % (unique_database, table_name)
- query = 'create table %s ' \
- '(s string, i int, b boolean, bi bigint, ts timestamp, dt date, ' \
- 'dc decimal(10, 3)) ' \
- 'stored as iceberg' \
- % qualified_table_name
+ query = """create table {}
+ (s string, i int, b boolean, bi bigint, ts timestamp, dt date,
+ dc decimal(10, 3))
+ stored as iceberg
+ tblproperties ('format-version'='{}')""".format(qualified_table_name, version)
self.client.execute(query)
-
# Insert data
# 1st data file:
query = 'insert into %s values ' \
(impala) 01/02: IMPALA-12802: Support ALTER TABLE for JDBC tables
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit eb5c8d6884f1c1bb3de66809e9ec59173072d5e8
Author: wzhou-code <wz...@cloudera.com>
AuthorDate: Tue Feb 27 11:56:05 2024 -0800
IMPALA-12802: Support ALTER TABLE for JDBC tables
IMPALA-12793 changes the syntax for creating JDBC table. The
configurations of connection credentials - url, username, password,
jdbc driver, etc, are set as table properties.
This patch allows user to change these table properties, or edit
columns via ALTER TABLE statement.
Testing:
- Added frontend analysis unit-tests.
- Added end-to-end unit-test.
- Passed Core tests
Change-Id: I5ebb5de2c686d2015db78641f78299dd5f33621e
Reviewed-on: http://gerrit.cloudera.org:8080/21088
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../impala/analysis/AlterTableAddColsStmt.java | 10 ++
.../AlterTableAddDropRangePartitionStmt.java | 3 +
.../analysis/AlterTableAddPartitionStmt.java | 8 ++
.../impala/analysis/AlterTableAlterColStmt.java | 12 ++
.../impala/analysis/AlterTableDropColStmt.java | 3 +
.../analysis/AlterTableDropPartitionStmt.java | 8 ++
.../AlterTableExecuteExpireSnapshotsStmt.java | 3 +
.../analysis/AlterTableExecuteRollbackStmt.java | 3 +
.../impala/analysis/AlterTableExecuteStmt.java | 3 +
.../analysis/AlterTableOrViewRenameStmt.java | 3 +
.../analysis/AlterTableOrViewSetOwnerStmt.java | 3 +
.../analysis/AlterTableRecoverPartitionsStmt.java | 3 +
.../impala/analysis/AlterTableReplaceColsStmt.java | 3 +
.../impala/analysis/AlterTableSetCachedStmt.java | 5 +
.../impala/analysis/AlterTableSetColumnStats.java | 3 +
.../analysis/AlterTableSetFileFormatStmt.java | 3 +
.../impala/analysis/AlterTableSetLocationStmt.java | 3 +
.../impala/analysis/AlterTableSetOwnerStmt.java | 3 +
.../analysis/AlterTableSetPartitionSpecStmt.java | 3 +
.../analysis/AlterTableSetRowFormatStmt.java | 3 +
.../apache/impala/analysis/AlterTableSetStmt.java | 4 +-
.../analysis/AlterTableSetTblProperties.java | 44 ++++++-
.../impala/analysis/AlterTableSortByStmt.java | 3 +
.../org/apache/impala/analysis/AlterTableStmt.java | 17 ++-
.../analysis/AlterTableUnSetTblProperties.java | 32 +++++
.../impala/analysis/AlterViewSetOwnerStmt.java | 3 +
.../impala/analysis/AlterViewSetTblProperties.java | 3 +
.../analysis/AlterViewUnSetTblProperties.java | 3 +
.../org/apache/impala/catalog/DataSourceTable.java | 18 +++
.../apache/impala/catalog/FeDataSourceTable.java | 1 +
.../impala/catalog/local/LocalDataSourceTable.java | 6 +
.../org/apache/impala/analysis/AnalyzeDDLTest.java | 145 +++++++++++++++++----
.../queries/QueryTest/jdbc-data-source.test | 120 +++++++++++++++++
33 files changed, 452 insertions(+), 35 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java
index dece3c4f8..f23d15166 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java
@@ -21,6 +21,8 @@ import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.impala.catalog.Column;
+import org.apache.impala.catalog.DataSourceTable;
+import org.apache.impala.catalog.FeDataSourceTable;
import org.apache.impala.catalog.FeHBaseTable;
import org.apache.impala.catalog.FeKuduTable;
import org.apache.impala.catalog.FeTable;
@@ -54,6 +56,9 @@ public class AlterTableAddColsStmt extends AlterTableStmt {
columnDefs_ = Lists.newArrayList(columnDefs);
}
+ @Override
+ public String getOperation() { return "ADD COLUMNS"; }
+
@Override
public void analyze(Analyzer analyzer) throws AnalysisException {
super.analyze(analyzer);
@@ -115,6 +120,11 @@ public class AlterTableAddColsStmt extends AlterTableStmt {
} else if (c.hasKuduOptions()) {
throw new AnalysisException("The specified column options are only supported " +
"in Kudu tables: " + c.toString());
+ } else if (t instanceof FeDataSourceTable) {
+ if (!DataSourceTable.isSupportedColumnType(c.getType())) {
+ throw new AnalysisException("Tables stored by JDBC do not support the " +
+ "column type: " + c.getType());
+ }
}
}
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java
index 64fbf3be3..e0f76cfa5 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java
@@ -63,6 +63,9 @@ public class AlterTableAddDropRangePartitionStmt extends AlterTableStmt {
operation_ = op;
}
+ @Override
+ public String getOperation() { return operation_.name().toUpperCase(); }
+
@Override
public String toSql(ToSqlOptions options) {
StringBuilder sb = new StringBuilder("ALTER TABLE " + getTbl());
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java
index 4ea4b9353..1f3919eed 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java
@@ -67,6 +67,14 @@ public class AlterTableAddPartitionStmt extends AlterTableStmt {
public boolean getIfNotExists() { return ifNotExists_; }
+ @Override
+ public String getOperation() {
+ StringBuilder sb = new StringBuilder("ADD ");
+ if (ifNotExists_) sb.append("IF NOT EXISTS ");
+ sb.append("PARTITION");
+ return sb.toString();
+ }
+
@Override
public String toSql(ToSqlOptions options) {
StringBuilder sb = new StringBuilder("ALTER TABLE ");
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java
index 6293c36f7..8c7308fc1 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java
@@ -21,6 +21,8 @@ import java.util.HashMap;
import java.util.Map;
import org.apache.impala.catalog.Column;
+import org.apache.impala.catalog.DataSourceTable;
+import org.apache.impala.catalog.FeDataSourceTable;
import org.apache.impala.catalog.FeHBaseTable;
import org.apache.impala.catalog.FeIcebergTable;
import org.apache.impala.catalog.FeKuduTable;
@@ -87,6 +89,9 @@ public class AlterTableAlterColStmt extends AlterTableStmt {
public String getColName() { return colName_; }
public ColumnDef getNewColDef() { return newColDef_; }
+ @Override
+ public String getOperation() { return "CHANGE COLUMN"; }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
@@ -187,5 +192,12 @@ public class AlterTableAlterColStmt extends AlterTableStmt {
"is not supported for complex types in Iceberg tables."));
}
}
+
+ if (t instanceof FeDataSourceTable) {
+ if (!DataSourceTable.isSupportedColumnType(newColDef_.getType())) {
+ throw new AnalysisException("Tables stored by JDBC do not support the " +
+ "column type: " + newColDef_.getType());
+ }
+ }
}
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java
index 8b9e875c3..fb7b14aae 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java
@@ -51,6 +51,9 @@ public class AlterTableDropColStmt extends AlterTableStmt {
return params;
}
+ @Override
+ public String getOperation() { return "DROP COLUMN"; }
+
@Override
public void analyze(Analyzer analyzer) throws AnalysisException {
super.analyze(analyzer);
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java
index 8e75f077f..a3b40a998 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java
@@ -79,6 +79,14 @@ public class AlterTableDropPartitionStmt extends AlterTableStmt {
public boolean getIfNotExists() { return ifExists_; }
+ @Override
+ public String getOperation() {
+ StringBuilder sb = new StringBuilder("DROP ");
+ if (ifExists_) sb.append("IF EXISTS ");
+ sb.append("PARTITION");
+ return sb.toString();
+ }
+
@Override
public String toSql(ToSqlOptions options) {
StringBuilder sb = new StringBuilder("ALTER TABLE " + getTbl());
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteExpireSnapshotsStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteExpireSnapshotsStmt.java
index 16d8a3753..a51621daa 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteExpireSnapshotsStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteExpireSnapshotsStmt.java
@@ -45,6 +45,9 @@ public class AlterTableExecuteExpireSnapshotsStmt extends AlterTableExecuteStmt
super(tableName, fnCallExpr);
}
+ @Override
+ public String getOperation() { return "EXECUTE EXPIRE_SNAPSHOTS"; }
+
@Override
public void analyze(Analyzer analyzer) throws AnalysisException {
super.analyze(analyzer);
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java
index ea4da0b67..0ddb57cbc 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java
@@ -53,6 +53,9 @@ public class AlterTableExecuteRollbackStmt extends AlterTableExecuteStmt {
super(tableName, fnCallExpr);
}
+ @Override
+ public String getOperation() { return "EXECUTE ROLLBACK"; }
+
@Override
public void analyze(Analyzer analyzer) throws AnalysisException {
super.analyze(analyzer);
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteStmt.java
index 0a0e33206..3ae47dcf0 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteStmt.java
@@ -41,6 +41,9 @@ public class AlterTableExecuteStmt extends AlterTableStmt {
fnCallExpr_ = (FunctionCallExpr) fnCallExpr;
}
+ @Override
+ public String getOperation() { return "EXECUTE"; }
+
/**
* Return an instance of a subclass of AlterTableExecuteStmt that can analyze the
* execute statement for the function call expression in 'expr'.
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java
index ba0c90a11..65265e353 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java
@@ -56,6 +56,9 @@ public class AlterTableOrViewRenameStmt extends AlterTableStmt {
return newDbName_;
}
+ @Override
+ public String getOperation() { return "RENAME"; }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewSetOwnerStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewSetOwnerStmt.java
index 25d9ecb12..cd052698e 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewSetOwnerStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewSetOwnerStmt.java
@@ -43,6 +43,9 @@ public abstract class AlterTableOrViewSetOwnerStmt extends AlterTableStmt {
owner_ = owner;
}
+ @Override
+ public String getOperation() { return "SET OWNER"; }
+
@Override
public void analyze(Analyzer analyzer) throws AnalysisException {
String ownerName = owner_.getOwnerName();
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableRecoverPartitionsStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableRecoverPartitionsStmt.java
index fe2b617fd..5fd2c3185 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableRecoverPartitionsStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableRecoverPartitionsStmt.java
@@ -32,6 +32,9 @@ public class AlterTableRecoverPartitionsStmt extends AlterTableStmt {
super(tableName);
}
+ @Override
+ public String getOperation() { return "RECOVER PARTITIONS"; }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java
index aeeca2431..8f39b91ea 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java
@@ -46,6 +46,9 @@ public class AlterTableReplaceColsStmt extends AlterTableStmt {
columnDefs_ = Lists.newArrayList(columnDefs);
}
+ @Override
+ public String getOperation() { return "REPLACE COLUMNS"; }
+
@Override
public void analyze(Analyzer analyzer) throws AnalysisException {
super.analyze(analyzer);
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java
index 0bcd2bf09..0cc82247c 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java
@@ -41,6 +41,11 @@ public class AlterTableSetCachedStmt extends AlterTableSetStmt {
cacheOp_ = cacheOp;
}
+ @Override
+ public String getOperation() {
+ return cacheOp_.shouldCache() ? "SET CACHED" : "SET UNCACHED";
+ }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetColumnStats.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetColumnStats.java
index 7aa6f1ff6..b642f44c3 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetColumnStats.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetColumnStats.java
@@ -53,6 +53,9 @@ public class AlterTableSetColumnStats extends AlterTableStmt {
statsMap_ = statsMap;
}
+ @Override
+ public String getOperation() { return "SET COLUMN STATS"; }
+
@Override
public void analyze(Analyzer analyzer) throws AnalysisException {
super.analyze(analyzer);
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java
index a45da9563..488fbc215 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java
@@ -40,6 +40,9 @@ public class AlterTableSetFileFormatStmt extends AlterTableSetStmt {
public THdfsFileFormat getFileFormat() { return fileFormat_; }
+ @Override
+ public String getOperation() { return "SET FILEFORMAT"; }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
index 01898d100..ac198df4a 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
@@ -55,6 +55,9 @@ public class AlterTableSetLocationStmt extends AlterTableSetStmt {
public HdfsUri getLocation() { return location_; }
+ @Override
+ public String getOperation() { return "SET LOCATION"; }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetOwnerStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetOwnerStmt.java
index 5932affed..c1424e8fc 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetOwnerStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetOwnerStmt.java
@@ -27,6 +27,9 @@ public class AlterTableSetOwnerStmt extends AlterTableOrViewSetOwnerStmt {
super(tableName, owner);
}
+ @Override
+ public String getOperation() { return "SET OWNER"; }
+
@Override
protected void validateType(TableRef tableRef) throws AnalysisException {
if (tableRef instanceof InlineViewRef) {
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
index 39d6a9728..b2cbf436c 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
@@ -43,6 +43,9 @@ public class AlterTableSetPartitionSpecStmt extends AlterTableStmt {
icebergPartSpec_ = icebergPartSpec;
}
+ @Override
+ public String getOperation() { return "SET PARTITION SPEC"; }
+
@Override
public String toSql(ToSqlOptions options) {
StringBuilder sb = new StringBuilder("ALTER TABLE ");
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetRowFormatStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetRowFormatStmt.java
index 2dcce78c8..56fc84766 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetRowFormatStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetRowFormatStmt.java
@@ -42,6 +42,9 @@ public class AlterTableSetRowFormatStmt extends AlterTableSetStmt {
public RowFormat getRowFormat() { return rowFormat_; }
+ @Override
+ public String getOperation() { return "SET ROW FORMAT"; }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java
index 4d6a10aca..6a3713907 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java
@@ -25,10 +25,10 @@ import org.apache.impala.common.AnalysisException;
/**
* Base class for all ALTER TABLE ... SET statements
*/
-public class AlterTableSetStmt extends AlterTableStmt {
+public abstract class AlterTableSetStmt extends AlterTableStmt {
protected final PartitionSet partitionSet_;
- public AlterTableSetStmt(TableName tableName, PartitionSet partitionSet) {
+ protected AlterTableSetStmt(TableName tableName, PartitionSet partitionSet) {
super(tableName);
partitionSet_ = partitionSet;
if (partitionSet_ != null) partitionSet_.setTableName(tableName);
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
index 46d6d271e..9191c41f4 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
@@ -28,6 +28,8 @@ import org.apache.iceberg.DataFile;
import org.apache.iceberg.mr.Catalogs;
import org.apache.iceberg.mr.InputFormatConfig;
import org.apache.impala.authorization.AuthorizationConfig;
+import org.apache.impala.catalog.DataSourceTable;
+import org.apache.impala.catalog.FeDataSourceTable;
import org.apache.impala.catalog.FeFsTable;
import org.apache.impala.catalog.FeHBaseTable;
import org.apache.impala.catalog.FeIcebergTable;
@@ -75,6 +77,12 @@ public class AlterTableSetTblProperties extends AlterTableSetStmt {
public Map<String, String> getTblProperties() { return tblProperties_; }
+ @Override
+ public String getOperation() {
+ return (targetProperty_ == TTablePropertyType.TBL_PROPERTY)
+ ? "SET TBLPROPERTIES" : "SET SERDEPROPERTIES";
+ }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
@@ -106,6 +114,8 @@ public class AlterTableSetTblProperties extends AlterTableSetStmt {
analyzeKuduTable(analyzer);
} else if (getTargetTable() instanceof FeIcebergTable) {
analyzeIcebergTable(analyzer);
+ } else if (getTargetTable() instanceof FeDataSourceTable) {
+ analyzeDataSourceTable(analyzer);
}
// Check avro schema when it is set in avro.schema.url or avro.schema.literal to
@@ -208,6 +218,28 @@ public class AlterTableSetTblProperties extends AlterTableSetStmt {
}
}
+ private void analyzeDataSourceTable(Analyzer analyzer) throws AnalysisException {
+ if (partitionSet_ != null) {
+ throw new AnalysisException("Partition is not supported for DataSource table.");
+ } else if (targetProperty_ == TTablePropertyType.SERDE_PROPERTY) {
+ throw new AnalysisException("ALTER TABLE SET SERDEPROPERTIES is not supported " +
+ "for DataSource table.");
+ }
+ // Cannot change internal properties of DataSource.
+ dataSourcePropertyCheck(DataSourceTable.TBL_PROP_DATA_SRC_NAME);
+ dataSourcePropertyCheck(DataSourceTable.TBL_PROP_INIT_STRING);
+ dataSourcePropertyCheck(DataSourceTable.TBL_PROP_LOCATION);
+ dataSourcePropertyCheck(DataSourceTable.TBL_PROP_CLASS);
+ dataSourcePropertyCheck(DataSourceTable.TBL_PROP_API_VER);
+ }
+
+ private void dataSourcePropertyCheck(String property) throws AnalysisException {
+ if (tblProperties_.containsKey(property)) {
+ throw new AnalysisException(String.format("Changing the '%s' table property is " +
+ "not supported for DataSource table.", property));
+ }
+ }
+
/**
* Check that Avro schema provided in avro.schema.url or avro.schema.literal is valid
* Json and contains only supported Impala types. If both properties are set, then
@@ -279,10 +311,14 @@ public class AlterTableSetTblProperties extends AlterTableSetStmt {
boolean containsSortingColumnProperties = tblProperties
.containsKey(AlterTableSortByStmt.TBL_PROP_SORT_COLUMNS);
- if ((containsOrderingProperties || containsSortingColumnProperties) &&
- table instanceof FeKuduTable) {
- throw new AnalysisException("'sort.*' table properties are not "
- + "supported for Kudu tables.");
+ if (containsOrderingProperties || containsSortingColumnProperties) {
+ if (table instanceof FeKuduTable) {
+ throw new AnalysisException("'sort.*' table properties are not "
+ + "supported for Kudu tables.");
+ } else if (table instanceof FeDataSourceTable) {
+ throw new AnalysisException("'sort.*' table properties are not "
+ + "supported for DataSource tables.");
+ }
}
TSortingOrder sortingOrder = TSortingOrder.LEXICAL;
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java
index 1a07a5d1d..576ca3289 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java
@@ -53,6 +53,9 @@ public class AlterTableSortByStmt extends AlterTableStmt {
sortingOrder_ = sortingOrder;
}
+ @Override
+ public String getOperation() { return "SORT BY"; }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
index 249987ed2..c5aa16a4a 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
@@ -43,6 +43,11 @@ public abstract class AlterTableStmt extends StatementBase {
table_ = null;
}
+ /*
+ * Returns the operation name of the ALTER TABLE statement.
+ */
+ public abstract String getOperation();
+
public String getTbl() { return tableName_.getTbl(); }
/**
@@ -101,10 +106,16 @@ public abstract class AlterTableStmt extends StatementBase {
// column statistics.
checkTransactionalTable();
if (table_ instanceof FeDataSourceTable
- && !(this instanceof AlterTableSetColumnStats)) {
+ && !(this instanceof AlterTableSetColumnStats
+ || this instanceof AlterTableAddColsStmt
+ || this instanceof AlterTableAlterColStmt
+ || this instanceof AlterTableDropColStmt
+ || this instanceof AlterTableSetTblProperties
+ || this instanceof AlterTableUnSetTblProperties)) {
+ boolean storedByJdbc = ((FeDataSourceTable)table_).isJdbcDataSourceTable();
throw new AnalysisException(String.format(
- "ALTER TABLE not allowed on a table produced by a data source: %s",
- tableName_));
+ "ALTER TABLE %s not allowed on a table %s: %s", getOperation(),
+ (storedByJdbc ? "STORED BY JDBC": "PRODUCED BY DATA SOURCE"), tableName_));
}
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableUnSetTblProperties.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableUnSetTblProperties.java
index 4d1d28da8..183cf4ef0 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableUnSetTblProperties.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableUnSetTblProperties.java
@@ -19,6 +19,8 @@ package org.apache.impala.analysis;
import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
import org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils;
+import org.apache.impala.catalog.DataSourceTable;
+import org.apache.impala.catalog.FeDataSourceTable;
import org.apache.impala.catalog.FeIcebergTable;
import org.apache.impala.catalog.FeKuduTable;
import org.apache.impala.catalog.IcebergTable;
@@ -58,6 +60,12 @@ public class AlterTableUnSetTblProperties extends AlterTableStmt {
public List<String> getTblPropertyKeys() { return tblPropertyKeys_; }
+ @Override
+ public String getOperation() {
+ return (targetProperty_ == TTablePropertyType.TBL_PROPERTY)
+ ? "UNSET TBLPROPERTIES" : "UNSET SERDEPROPERTIES";
+ }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = super.toThrift();
@@ -93,6 +101,8 @@ public class AlterTableUnSetTblProperties extends AlterTableStmt {
analyzeKuduTable(analyzer);
} else if (getTargetTable() instanceof FeIcebergTable) {
analyzeIcebergTable(analyzer);
+ } else if (getTargetTable() instanceof FeDataSourceTable) {
+ analyzeDataSourceTable(analyzer);
}
// Unsetting avro.schema.url or avro.schema.literal are not allowed to
@@ -128,6 +138,28 @@ public class AlterTableUnSetTblProperties extends AlterTableStmt {
propertyCheck(IcebergTable.METADATA_LOCATION, "Iceberg");
}
+ private void analyzeDataSourceTable(Analyzer analyzer) throws AnalysisException {
+ if (partitionSet_ != null) {
+ throw new AnalysisException("Partition is not supported for DataSource table.");
+ } else if (targetProperty_ == TTablePropertyType.SERDE_PROPERTY) {
+ throw new AnalysisException("ALTER TABLE UNSET SERDEPROPERTIES is not supported " +
+ "for DataSource table.");
+ }
+ // Cannot unset internal properties of DataSource.
+ propertyCheck(DataSourceTable.TBL_PROP_DATA_SRC_NAME, "DataSource");
+ propertyCheck(DataSourceTable.TBL_PROP_INIT_STRING, "DataSource");
+ propertyCheck(DataSourceTable.TBL_PROP_LOCATION, "DataSource");
+ propertyCheck(DataSourceTable.TBL_PROP_CLASS, "DataSource");
+ propertyCheck(DataSourceTable.TBL_PROP_API_VER, "DataSource");
+ // Cannot unset properties which are required JDBC parameters.
+ for (String property : tblPropertyKeys_) {
+ if (DataSourceTable.isRequiredJdbcParameter(property)) {
+ throw new AnalysisException(String.format("Unsetting the '%s' table property " +
+ "is not supported for JDBC DataSource table.", property));
+ }
+ }
+ }
+
private void propertyCheck(String property, String tableType) throws AnalysisException {
if (tblPropertyKeys_.contains(property)) {
throw new AnalysisException(String.format("Unsetting the '%s' table property is " +
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterViewSetOwnerStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterViewSetOwnerStmt.java
index 4e1fc52f5..b544131ed 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterViewSetOwnerStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterViewSetOwnerStmt.java
@@ -27,6 +27,9 @@ public class AlterViewSetOwnerStmt extends AlterTableOrViewSetOwnerStmt {
super(tableName, owner);
}
+ @Override
+ public String getOperation() { return "SET OWNER"; }
+
@Override
protected void validateType(TableRef tableRef) throws AnalysisException {
if (!(tableRef instanceof InlineViewRef)) {
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterViewSetTblProperties.java b/fe/src/main/java/org/apache/impala/analysis/AlterViewSetTblProperties.java
index 85408ca05..701e092be 100755
--- a/fe/src/main/java/org/apache/impala/analysis/AlterViewSetTblProperties.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterViewSetTblProperties.java
@@ -47,6 +47,9 @@ public class AlterViewSetTblProperties extends AlterTableSetStmt {
CreateTableStmt.unescapeProperties(tblProperties_);
}
+ @Override
+ public String getOperation() { return "SET TBLPROPERTIES"; }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = new TAlterTableParams();
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterViewUnSetTblProperties.java b/fe/src/main/java/org/apache/impala/analysis/AlterViewUnSetTblProperties.java
index ea6cb4617..6e42f9fc2 100755
--- a/fe/src/main/java/org/apache/impala/analysis/AlterViewUnSetTblProperties.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterViewUnSetTblProperties.java
@@ -48,6 +48,9 @@ public class AlterViewUnSetTblProperties extends AlterTableStmt {
ifExists_ = ifExist;
}
+ @Override
+ public String getOperation() { return "UNSET TBLPROPERTIES"; }
+
@Override
public TAlterTableParams toThrift() {
TAlterTableParams params = new TAlterTableParams();
diff --git a/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java b/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
index 642ca481c..2be5411ae 100644
--- a/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
@@ -114,6 +114,12 @@ public class DataSourceTable extends Table implements FeDataSourceTable {
@Override // FeDataSourceTable
public int getNumNodes() { return 1; }
+ @Override // FeDataSourceTable
+ public boolean isJdbcDataSourceTable() {
+ return (dataSource_ != null && dataSource_.name != null &&
+ dataSource_.name.equals(IMPALA_BUILTIN_JDBC_DATASOURCE));
+ }
+
@Override
public TCatalogObjectType getCatalogObjectType() { return TCatalogObjectType.TABLE; }
@@ -190,6 +196,18 @@ public class DataSourceTable extends Table implements FeDataSourceTable {
}
}
+ /**
+ * Check if the given property name is one of required JDBC/DBCP parameters.
+ */
+ public static boolean isRequiredJdbcParameter(String propertyName) {
+ for (JdbcStorageConfig config : JdbcStorageConfig.values()) {
+ if (config.getPropertyName().equalsIgnoreCase(propertyName)) {
+ return config.isRequired();
+ }
+ }
+ return false;
+ }
+
/**
* Create columns corresponding to fieldSchemas.
* Throws a TableLoadingException if the metadata is incompatible with what we
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeDataSourceTable.java b/fe/src/main/java/org/apache/impala/catalog/FeDataSourceTable.java
index a76f38db1..4f21946e0 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeDataSourceTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeDataSourceTable.java
@@ -36,4 +36,5 @@ public interface FeDataSourceTable extends FeTable {
// in Frontend.getTableStats?
TResultSet getTableStats();
+ boolean isJdbcDataSourceTable();
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java b/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
index dea93bcc5..2d083d795 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
@@ -146,6 +146,12 @@ public class LocalDataSourceTable extends LocalTable implements FeDataSourceTabl
@Override // FeDataSourceTable
public int getNumNodes() { return 1; }
+ @Override // FeDataSourceTable
+ public boolean isJdbcDataSourceTable() {
+ return (dataSource_ != null && dataSource_.name != null &&
+ dataSource_.name.equals(DataSourceTable.IMPALA_BUILTIN_JDBC_DATASOURCE));
+ }
+
/**
* Returns statistics on this table as a tabular result set. Used for the
* SHOW TABLE STATS statement. The schema of the returned TResultSet is set
diff --git a/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java b/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
index f84bb342f..02a3aebdb 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
@@ -119,10 +119,15 @@ public class AnalyzeDDLTest extends FrontendTestBase {
AnalysisError("alter table functional.alltypes_view " + kw +
" partition(year=2050, month=10)",
"ALTER TABLE not allowed on a view: functional.alltypes_view");
+ // Cannot ALTER TABLE to add or drop partition for data source tables.
AnalysisError("alter table functional.alltypes_datasource " + kw +
" partition(year=2050, month=10)",
- "ALTER TABLE not allowed on a table produced by a data source: " +
- "functional.alltypes_datasource");
+ "ALTER TABLE " + kw.toUpperCase() + " PARTITION not allowed on a table " +
+ "PRODUCED BY DATA SOURCE: functional.alltypes_datasource");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource " + kw +
+ " partition(year=2050, month=10)",
+ "ALTER TABLE " + kw.toUpperCase() + " PARTITION not allowed on a table " +
+ "STORED BY JDBC: functional.alltypes_jdbc_datasource");
// NULL partition keys
AnalyzesOk("alter table functional.alltypes " + kw +
@@ -450,6 +455,10 @@ public class AnalyzeDDLTest extends FrontendTestBase {
// Valid unicode column name.
AnalyzesOk("alter table functional.alltypes add column `???` int");
+ // ALTER TABLE to add column for data source tables.
+ AnalyzesOk("alter table functional.alltypes_datasource add column c1 string");
+ AnalyzesOk("alter table functional.alltypes_jdbc_datasource add column c1 string");
+
// Column name must be unique for add.
AnalysisError("alter table functional.alltypes add column int_col int",
"Column already exists: int_col");
@@ -478,10 +487,10 @@ public class AnalyzeDDLTest extends FrontendTestBase {
AnalysisError("alter table allcomplextypes.int_array_col add column c1 string",
createAnalysisCtx("functional"),
"ALTER TABLE not allowed on a nested collection: allcomplextypes.int_array_col");
- // Cannot ALTER TABLE produced by a data source.
- AnalysisError("alter table functional.alltypes_datasource add column c1 string",
- "ALTER TABLE not allowed on a table produced by a data source: " +
- "functional.alltypes_datasource");
+ // Cannot ALTER TABLE to add column with unsupported data type for data source
+ // table.
+ AnalysisError("alter table functional.alltypes_jdbc_datasource add column i binary",
+ "Tables stored by JDBC do not support the column type: BINARY");
// Cannot ALTER TABLE ADD COLUMNS on an HBase table.
AnalysisError("alter table functional_hbase.alltypes add column i int",
@@ -523,6 +532,12 @@ public class AnalyzeDDLTest extends FrontendTestBase {
AnalyzesOk("alter table functional.alltypes add columns" +
"(`시스???पताED` int)");
+ // ALTER TABLE to add columns for data source tables.
+ AnalyzesOk("alter table functional.alltypes_datasource add columns " +
+ "(c1 string comment 'hi')");
+ AnalyzesOk("alter table functional.alltypes_jdbc_datasource add columns " +
+ "(c1 string comment 'hi')");
+
// Column name must be unique for add.
AnalysisError("alter table functional.alltypes add columns (int_col int)",
"Column already exists: int_col");
@@ -557,11 +572,11 @@ public class AnalyzeDDLTest extends FrontendTestBase {
"add columns (c1 string comment 'hi')",
createAnalysisCtx("functional"),
"ALTER TABLE not allowed on a nested collection: allcomplextypes.int_array_col");
- // Cannot ALTER TABLE produced by a data source.
- AnalysisError("alter table functional.alltypes_datasource " +
- "add columns (c1 string comment 'hi')",
- "ALTER TABLE not allowed on a table produced by a data source: " +
- "functional.alltypes_datasource");
+ // Cannot ALTER TABLE to add columns with unsupported data type for data source
+ // table.
+ AnalysisError("alter table functional.alltypes_jdbc_datasource add columns " +
+ "(c1 binary comment 'hi')",
+ "Tables stored by JDBC do not support the column type: BINARY");
// Cannot ALTER TABLE ADD COLUMNS on an HBase table.
AnalysisError("alter table functional_hbase.alltypes add columns (i int)",
@@ -632,11 +647,15 @@ public class AnalyzeDDLTest extends FrontendTestBase {
"replace columns (c1 string comment 'hi')",
createAnalysisCtx("functional"),
"ALTER TABLE not allowed on a nested collection: allcomplextypes.int_array_col");
- // Cannot ALTER TABLE produced by a data source.
+ // Cannot ALTER TABLE to replace columns for data source tables.
AnalysisError("alter table functional.alltypes_datasource " +
"replace columns (c1 string comment 'hi')",
- "ALTER TABLE not allowed on a table produced by a data source: " +
+ "ALTER TABLE REPLACE COLUMNS not allowed on a table PRODUCED BY DATA SOURCE: " +
"functional.alltypes_datasource");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource " +
+ "replace columns (c1 string comment 'hi')",
+ "ALTER TABLE REPLACE COLUMNS not allowed on a table STORED BY JDBC: " +
+ "functional.alltypes_jdbc_datasource");
// Cannot ALTER TABLE REPLACE COLUMNS on a HBase table.
AnalysisError("alter table functional_hbase.alltypes replace columns (i int)",
@@ -651,6 +670,10 @@ public class AnalyzeDDLTest extends FrontendTestBase {
public void TestAlterTableDropColumn() throws AnalysisException {
AnalyzesOk("alter table functional.alltypes drop column int_col");
+ // ALTER TABLE to drop column for data source tables.
+ AnalyzesOk("alter table functional.alltypes_datasource drop column int_col");
+ AnalyzesOk("alter table functional.alltypes_jdbc_datasource drop column int_col");
+
AnalysisError("alter table functional.alltypes drop column no_col",
"Column 'no_col' does not exist in table: functional.alltypes");
@@ -675,10 +698,6 @@ public class AnalyzeDDLTest extends FrontendTestBase {
AnalysisError("alter table allcomplextypes.int_array_col drop column int_col",
createAnalysisCtx("functional"),
"ALTER TABLE not allowed on a nested collection: allcomplextypes.int_array_col");
- // Cannot ALTER TABLE produced by a data source.
- AnalysisError("alter table functional.alltypes_datasource drop column int_col",
- "ALTER TABLE not allowed on a table produced by a data source: " +
- "functional.alltypes_datasource");
// Cannot ALTER TABLE DROP COLUMN on an HBase table.
AnalysisError("alter table functional_hbase.alltypes drop column int_col",
@@ -699,6 +718,15 @@ public class AnalyzeDDLTest extends FrontendTestBase {
AnalyzesOk("alter table functional.alltypes change int_col int_col int comment 'c'");
AnalyzesOk("alter table functional.alltypes change column int_col `汉字` int");
+ // ALTER TABLE to change column for data source tables.
+ AnalyzesOk("alter table functional.alltypes_datasource " +
+ "change column int_col int_col2 int");
+ AnalyzesOk("alter table functional.alltypes_jdbc_datasource " +
+ "change column int_col int_col2 int");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource " +
+ "change column int_col bin_col binary",
+ "Tables stored by JDBC do not support the column type: BINARY");
+
AnalysisError("alter table functional.alltypes change column no_col c1 int",
"Column 'no_col' does not exist in table: functional.alltypes");
@@ -709,7 +737,6 @@ public class AnalyzeDDLTest extends FrontendTestBase {
"alter table functional.alltypes change column int_col Tinyint_col int",
"Column already exists: tinyint_col");
-
// Table/Db does not exist
AnalysisError("alter table db_does_not_exist.alltypes change c1 c2 int",
"Could not resolve table reference: 'db_does_not_exist.alltypes'");
@@ -725,11 +752,6 @@ public class AnalyzeDDLTest extends FrontendTestBase {
"change column int_col int_col2 int",
createAnalysisCtx("functional"),
"ALTER TABLE not allowed on a nested collection: allcomplextypes.int_array_col");
- // Cannot ALTER TABLE produced by a data source.
- AnalysisError("alter table functional.alltypes_datasource " +
- "change column int_col int_col2 int",
- "ALTER TABLE not allowed on a table produced by a data source: " +
- "functional.alltypes_datasource");
// Cannot ALTER TABLE CHANGE COLUMN on an HBase table.
AnalysisError("alter table functional_hbase.alltypes CHANGE COLUMN int_col i int",
@@ -762,6 +784,11 @@ public class AnalyzeDDLTest extends FrontendTestBase {
"ALTER TABLE SET ROW FORMAT is only supported on TEXT or SEQUENCE file formats");
AnalyzesOk("alter table functional.alltypesmixedformat partition(year=2009,month=1) " +
"set row format delimited fields terminated by ' '");
+ // Cannot ALTER TABLE to set row format for data source tables.
+ AnalysisError("alter table functional.alltypes_jdbc_datasource set row format " +
+ "delimited fields terminated by ' '",
+ "ALTER TABLE SET ROW FORMAT not allowed on a table STORED BY JDBC: " +
+ "functional.alltypes_jdbc_datasource");
}
@Test
@@ -796,6 +823,39 @@ public class AnalyzeDDLTest extends FrontendTestBase {
AnalyzesOk("alter table functional.alltypes PARTITION (year<=2010, month=11) " +
"set serdeproperties ('a'='2')");
+ // ALTER TABLE to set or unset tblproperties for data source tables.
+ AnalyzesOk("alter table functional.alltypes_datasource set tblproperties ('a'='2')");
+ AnalyzesOk("alter table functional.alltypes_jdbc_datasource set tblproperties " +
+ "('a'='2')");
+ AnalyzesOk("alter table functional.alltypes_datasource unset tblproperties " +
+ "if exists ('a')");
+ AnalyzesOk("alter table functional.alltypes_jdbc_datasource unset tblproperties " +
+ "if exists ('a')");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource set " +
+ "serdeproperties ('a'='2')",
+ "ALTER TABLE SET SERDEPROPERTIES is not supported for DataSource table");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource PARTITION " +
+ "(year=2010) set tblproperties ('a'='2')",
+ "Table is not partitioned: functional.alltypes_jdbc_datasource");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource set tblproperties " +
+ "('__IMPALA_DATA_SOURCE_NAME'='test')",
+ "Changing the '__IMPALA_DATA_SOURCE_NAME' table property is not supported for " +
+ "DataSource table.");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource unset " +
+ "serdeproperties ('a')",
+ "ALTER TABLE UNSET SERDEPROPERTIES is not supported for DataSource table");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource PARTITION " +
+ "(year=2010) unset tblproperties ('a')",
+ "Partition is not supported for DataSource table.");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource unset " +
+ "tblproperties ('__IMPALA_DATA_SOURCE_NAME')",
+ "Unsetting the '__IMPALA_DATA_SOURCE_NAME' table property is not supported " +
+ "for DataSource table.");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource unset " +
+ "tblproperties ('driver.url')",
+ "Unsetting the 'driver.url' table property is not supported for JDBC " +
+ "DataSource table.");
+
AnalyzesOk("alter table functional.alltypes set tblproperties('sort.columns'='id')");
AnalyzesOk("alter table functional.alltypes set tblproperties(" +
"'sort.columns'='INT_COL,id')");
@@ -966,6 +1026,16 @@ public class AnalyzeDDLTest extends FrontendTestBase {
AnalyzesOk("alter table functional.alltypes set location " +
"'file:///test-warehouse/a/b'");
+ // Cannot ALTER TABLE to set location for data source tables.
+ AnalysisError("alter table functional.alltypes_datasource set location " +
+ "'file:///test-warehouse/a/b'",
+ "ALTER TABLE SET LOCATION not allowed on a table PRODUCED BY DATA SOURCE: " +
+ "functional.alltypes_datasource");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource set location " +
+ "'file:///test-warehouse/a/b'",
+ "ALTER TABLE SET LOCATION not allowed on a table STORED BY JDBC: " +
+ "functional.alltypes_jdbc_datasource");
+
// Invalid location
AnalysisError("alter table functional.alltypes set location 'test/warehouse'",
"URI path must be absolute: test/warehouse");
@@ -983,10 +1053,14 @@ public class AnalyzeDDLTest extends FrontendTestBase {
AnalysisError("alter table allcomplextypes.int_array_col set fileformat sequencefile",
createAnalysisCtx("functional"),
"ALTER TABLE not allowed on a nested collection: allcomplextypes.int_array_col");
- // Cannot ALTER TABLE produced by a data source.
+ // Cannot ALTER TABLE to set fileformat for data source tables.
AnalysisError("alter table functional.alltypes_datasource set fileformat parquet",
- "ALTER TABLE not allowed on a table produced by a data source: " +
+ "ALTER TABLE SET FILEFORMAT not allowed on a table PRODUCED BY DATA SOURCE: " +
"functional.alltypes_datasource");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource set fileformat " +
+ "parquet",
+ "ALTER TABLE SET FILEFORMAT not allowed on a table STORED BY JDBC: " +
+ "functional.alltypes_jdbc_datasource");
// Cannot ALTER TABLE SET on an HBase table.
AnalysisError("alter table functional_hbase.alltypes set tblproperties('a'='b')",
@@ -1057,6 +1131,11 @@ public class AnalyzeDDLTest extends FrontendTestBase {
AnalysisError("alter table functional.alltypestiny partition(year=9999, month=1) " +
"set cached in 'testPool'",
"No matching partition(s) found.");
+ // Cannot ALTER TABLE to set cached for data source tables.
+ AnalysisError("alter table functional.alltypes_jdbc_datasource set cached in " +
+ "'testPool'",
+ "ALTER TABLE SET CACHED not allowed on a table STORED BY JDBC: " +
+ "functional.alltypes_jdbc_datasource");
}
@Test
@@ -1343,8 +1422,10 @@ public class AnalyzeDDLTest extends FrontendTestBase {
// It should be okay to rename an HBase table.
AnalyzesOk("alter table functional_hbase.alltypes rename to new_alltypes");
- // It should be okay to rename a table produced by a data source.
+ // It should be okay to rename data source tables.
AnalyzesOk("alter table functional.alltypes_datasource rename to new_datasrc_tbl");
+ AnalyzesOk("alter table functional.alltypes_jdbc_datasource rename to " +
+ "new_jdbc_datasrc_tbl");
}
@Test
@@ -1378,6 +1459,10 @@ public class AnalyzeDDLTest extends FrontendTestBase {
"SORT BY column 'foo' in table.");
AnalysisError("alter table functional_hbase.alltypes sort by (id, foo)",
"ALTER TABLE SORT BY not supported on HBase tables.");
+ // Cannot ALTER TABLE to sort by for data source tables.
+ AnalysisError("alter table functional.alltypes_jdbc_datasource sort by (id)",
+ "ALTER TABLE SORT BY not allowed on a table STORED BY JDBC: " +
+ "functional.alltypes_jdbc_datasource");
}
@Test
@@ -1396,6 +1481,10 @@ public class AnalyzeDDLTest extends FrontendTestBase {
"not find SORT BY column 'foo' in table.");
AnalysisError("alter table functional_hbase.alltypes sort by zorder (id, foo)",
"ALTER TABLE SORT BY not supported on HBase tables.");
+ // Cannot ALTER TABLE to sort by zorder for data source tables.
+ AnalysisError("alter table functional.alltypes_jdbc_datasource sort by zorder (id)",
+ "ALTER TABLE SORT BY not allowed on a table STORED BY JDBC: " +
+ "functional.alltypes_jdbc_datasource");
}
@Test
@@ -1648,6 +1737,10 @@ public class AnalyzeDDLTest extends FrontendTestBase {
"Unsupported column options for non-Kudu table: 'int_col INT COMPRESSION LZ4'");
AnalysisError("alter table functional.alltypes alter int_col drop default",
"Unsupported column option for non-Kudu table: DROP DEFAULT");
+ AnalysisError("alter table functional.alltypes_datasource alter int_col drop " +
+ "default", "Unsupported column option for non-Kudu table: DROP DEFAULT");
+ AnalysisError("alter table functional.alltypes_jdbc_datasource alter int_col drop " +
+ "default", "Unsupported column option for non-Kudu table: DROP DEFAULT");
}
ComputeStatsStmt checkComputeStatsStmt(String stmt) throws AnalysisException {
diff --git a/testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test b/testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test
index 84148d614..72e375b80 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test
@@ -433,3 +433,123 @@ DROP TABLE alltypes_jdbc_datasource_2;
---- RESULTS
'Table has been dropped.'
====
+---- QUERY
+# Create JDBC table with unknown database type.
+DROP TABLE IF EXISTS jdbc_test;
+CREATE TABLE jdbc_test (
+ id INT,
+ bool_col BOOLEAN,
+ int_col INT)
+STORED BY JDBC
+TBLPROPERTIES (
+"database.type"="UNKNOWN",
+"jdbc.url"="jdbc:postgresql://$INTERNAL_LISTEN_HOST:5432/functional",
+"jdbc.driver"="org.postgresql.Driver",
+"driver.url"="$FILESYSTEM_PREFIX/test-warehouse/data-sources/jdbc-drivers/postgresql-jdbc.jar",
+"dbcp.username"="hiveuser",
+"dbcp.password"="password",
+"table"="alltypes");
+---- RESULTS
+'Table has been created.'
+====
+---- QUERY
+select count(*) from jdbc_test;
+---- CATCH
+row_regex:.*InternalException:.*Unknown database type.
+====
+---- QUERY
+# Alter JDBC table to change database type as POSTGRES.
+ALTER TABLE jdbc_test
+SET TBLPROPERTIES ("database.type"="POSTGRES");
+---- RESULTS
+'Updated table.'
+====
+---- QUERY
+select count(*) from jdbc_test;
+---- RESULTS
+7300
+---- TYPES
+BIGINT
+====
+---- QUERY
+# Alter JDBC table to set wrong username and password.
+ALTER TABLE jdbc_test
+SET TBLPROPERTIES ("dbcp.username"="UNKNOWN", "dbcp.password"="NONE");
+---- RESULTS
+'Updated table.'
+====
+---- QUERY
+select * from jdbc_test order by id limit 5;
+---- CATCH
+row_regex:.*Caught exception.*(FATAL: password authentication failed for user "UNKNOWN")
+====
+---- QUERY
+# Alter JDBC table to set correct username and password.
+ALTER TABLE jdbc_test
+SET TBLPROPERTIES ("dbcp.username"="hiveuser", "dbcp.password"="password");
+---- RESULTS
+'Updated table.'
+====
+---- QUERY
+select * from jdbc_test order by id limit 5;
+---- RESULTS
+0,true,0
+1,false,1
+2,true,2
+3,false,3
+4,true,4
+---- TYPES
+INT, BOOLEAN, INT
+====
+---- QUERY
+# Alter JDBC table to unset the required JDBC configuration "table".
+ALTER TABLE jdbc_test UNSET TBLPROPERTIES ("table");
+---- CATCH
+row_regex:.*AnalysisException: Unsetting the 'table' table property is not supported for JDBC DataSource table
+====
+---- QUERY
+# Alter table to drop a column.
+ALTER TABLE jdbc_test DROP COLUMN int_col;
+---- RESULTS
+'Column has been dropped.'
+====
+---- QUERY
+# Alter table to add a new column.
+ALTER TABLE jdbc_test ADD COLUMN IF NOT EXISTS date_col DATE;
+---- RESULTS
+'New column(s) have been added to the table.'
+====
+---- QUERY
+select * from jdbc_test order by id limit 5;
+---- RESULTS
+0,true,2009-01-01
+1,false,2009-01-01
+2,true,2009-01-01
+3,false,2009-01-01
+4,true,2009-01-01
+---- TYPES
+INT, BOOLEAN, DATE
+====
+---- QUERY
+# Alter table to change a column.
+ALTER TABLE jdbc_test CHANGE COLUMN date_col timestamp_col TIMESTAMP;
+---- RESULTS
+'Column has been altered.'
+====
+---- QUERY
+select * from jdbc_test order by id limit 5;
+---- RESULTS
+0,true,2009-01-01 00:00:00
+1,false,2009-01-01 00:01:00
+2,true,2009-01-01 00:02:00.100000000
+3,false,2009-01-01 00:03:00.300000000
+4,true,2009-01-01 00:04:00.600000000
+---- TYPES
+INT, BOOLEAN, TIMESTAMP
+====
+---- QUERY
+# Drop table.
+DROP TABLE jdbc_test;
+---- RESULTS
+'Table has been dropped.'
+====