You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/09/27 17:12:38 UTC
[GitHub] [hive] ashish-kumar-sharma opened a new pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
ashish-kumar-sharma opened a new pull request #1528:
URL: https://github.com/apache/hive/pull/1528
… column names
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Compaction for tables where a bucketed column has been renamed fails since the list of bucketed columns in the StorageDescriptor doesn't get updated when the column is renamed, therefore we can't recreate the table correctly during compaction.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Update bucket column name in AlterTableChangeColumnOperation along with column name.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Unit test added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498436517
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
##########
@@ -1554,4 +1555,30 @@ public static Partition createMetaPartitionObject(Table tbl, Map<String, String>
}
return tpart;
}
+
+ /**
+ * Validate bucket columns should belong to table columns.
+ * @param sd StorageDescriptor of given table
+ * @return true if bucket columns are empty or belong to table columns else false
+ */
+ public static boolean validateBucketColumns(StorageDescriptor sd) {
+ if (CollectionUtils.isNotEmpty(sd.getBucketCols())) {
+ List<String> columnNames =
+ getColumnNames(sd.getCols()).stream().map(String::toLowerCase).collect(Collectors.toList());
+ return columnNames.containsAll(sd.getBucketCols().stream().map(String::toLowerCase).collect(Collectors.toList()));
Review comment:
Done
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
##########
@@ -63,6 +64,8 @@
import org.datanucleus.api.jdo.JDOPersistenceManagerFactory;
import org.junit.Assert;
import org.junit.Before;
+import org.junit.experimental.categories.Category;
Review comment:
removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498900786
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
##########
@@ -1554,4 +1555,32 @@ public static Partition createMetaPartitionObject(Table tbl, Map<String, String>
}
return tpart;
}
+
+ /**
+ * Validate bucket columns should belong to table columns.
+ * @param sd StorageDescriptor of given table
+ * @return true if bucket columns are empty or belong to table columns else false
+ */
+ public static boolean validateBucketColumns(StorageDescriptor sd) {
+ List<String> columnNames = getColumnNames(sd.getCols());
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498900535
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
##########
@@ -1554,4 +1555,32 @@ public static Partition createMetaPartitionObject(Table tbl, Map<String, String>
}
return tpart;
}
+
+ /**
+ * Validate bucket columns should belong to table columns.
+ * @param sd StorageDescriptor of given table
+ * @return true if bucket columns are empty or belong to table columns else false
+ */
+ public static boolean validateBucketColumns(StorageDescriptor sd) {
+ List<String> columnNames = getColumnNames(sd.getCols());
+ if(CollectionUtils.isNotEmpty(sd.getBucketCols()) && CollectionUtils.isNotEmpty(columnNames)){
+ return columnNames.containsAll(sd.getBucketCols().stream().map(String::toLowerCase).collect(Collectors.toList()));
+ } else if (CollectionUtils.isNotEmpty(sd.getBucketCols()) && CollectionUtils.isEmpty(columnNames)) {
+ return false;
+ } else {
+ return true;
+ }
+ }
+
+ /**
+ * Generate column name list from the fieldSchema list
+ * @param cols fieldSchema list
+ * @return column name list
+ */
+ public static List<String> getColumnNames(List<FieldSchema> cols) {
+ if (CollectionUtils.isNotEmpty(cols)) {
+ return cols.stream().map(FieldSchema::getName).collect(Collectors.toList());
Review comment:
Expected column name in lower case. But in order to be fail safe added toLowerCase() here also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498901990
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##########
@@ -130,6 +130,11 @@ public void alterTable(RawStore msdb, Warehouse wh, String catName, String dbnam
throw new InvalidOperationException("Invalid column " + validate);
}
+ // Validate bucketedColumns in new table
+ if (!MetaStoreServerUtils.validateBucketColumns(newt.getSd())) {
+ throw new InvalidOperationException("Bucket column doesn't match with any table columns");
Review comment:
1. Converted return type to List<String>.
2. Added Log.error() along with column name.
3. Added column to exception also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r497652090
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/change/AlterTableChangeColumnOperation.java
##########
@@ -33,6 +35,7 @@
import org.apache.hadoop.hive.ql.metadata.Partition;
import org.apache.hadoop.hive.ql.metadata.Table;
import org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils;
+import org.apache.spark.util.CollectionsUtils;
Review comment:
required?
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
##########
@@ -63,6 +64,8 @@
import org.datanucleus.api.jdo.JDOPersistenceManagerFactory;
import org.junit.Assert;
import org.junit.Before;
+import org.junit.experimental.categories.Category;
Review comment:
required?
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
##########
@@ -1554,4 +1555,30 @@ public static Partition createMetaPartitionObject(Table tbl, Map<String, String>
}
return tpart;
}
+
+ /**
+ * Validate bucket columns should belong to table columns.
+ * @param sd StorageDescriptor of given table
+ * @return true if bucket columns are empty or belong to table columns else false
+ */
+ public static boolean validateBucketColumns(StorageDescriptor sd) {
+ if (CollectionUtils.isNotEmpty(sd.getBucketCols())) {
+ List<String> columnNames =
+ getColumnNames(sd.getCols()).stream().map(String::toLowerCase).collect(Collectors.toList());
+ return columnNames.containsAll(sd.getBucketCols().stream().map(String::toLowerCase).collect(Collectors.toList()));
Review comment:
add a null check for columnNames? getColumnNames returns null too.
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
##########
@@ -3428,4 +3432,44 @@ public void testUpdatePartitionStat_doesNotUpdateStats() throws Exception {
m.invoke(hms, tbl, part, false, null);
verify(wh, never()).getFileStatusesForLocation(part.getSd().getLocation());
}
+
+ @Test(expected = InvalidOperationException.class)
Review comment:
Also, Is it a positive test case or negative? I see that there is an expected error but we are doing the positive validations too. We should break this into two, right?
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
##########
@@ -3428,4 +3432,44 @@ public void testUpdatePartitionStat_doesNotUpdateStats() throws Exception {
m.invoke(hms, tbl, part, false, null);
verify(wh, never()).getFileStatusesForLocation(part.getSd().getLocation());
}
+
+ @Test(expected = InvalidOperationException.class)
Review comment:
Won't adding a qtest be better?
You are explicitly doing `tblUpdated.getSd().getBucketCols()` and updating them in the test. qtest will check end to end flow and you don't need to bother about getting Sd and updating the cols explicitly (Just a single alter table will do the trick).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498440059
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/change/AlterTableChangeColumnOperation.java
##########
@@ -33,6 +35,7 @@
import org.apache.hadoop.hive.ql.metadata.Partition;
import org.apache.hadoop.hive.ql.metadata.Table;
import org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils;
+import org.apache.spark.util.CollectionsUtils;
Review comment:
Removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498900655
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
##########
@@ -1554,4 +1555,32 @@ public static Partition createMetaPartitionObject(Table tbl, Map<String, String>
}
return tpart;
}
+
+ /**
+ * Validate bucket columns should belong to table columns.
+ * @param sd StorageDescriptor of given table
+ * @return true if bucket columns are empty or belong to table columns else false
+ */
+ public static boolean validateBucketColumns(StorageDescriptor sd) {
+ List<String> columnNames = getColumnNames(sd.getCols());
+ if(CollectionUtils.isNotEmpty(sd.getBucketCols()) && CollectionUtils.isNotEmpty(columnNames)){
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh merged pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
sankarh merged pull request #1528:
URL: https://github.com/apache/hive/pull/1528
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498905028
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/change/AlterTableChangeColumnOperation.java
##########
@@ -72,6 +74,10 @@ protected void doAlteration(Table table, Partition partition) throws HiveExcepti
if (desc.getNewColumnComment() != null) {
oldColumn.setComment(desc.getNewColumnComment());
}
+ if (CollectionUtils.isNotEmpty(sd.getBucketCols()) && sd.getBucketCols().contains(oldColumnName)) {
+ sd.getBucketCols().remove(oldColumnName);
+ sd.getBucketCols().add(desc.getNewColumnName());
Review comment:
newColumnName is converted toLowerCase in query planning while populating "desc" but to be fail safe i have added toLowerCase() here also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498902660
##########
File path: ql/src/test/queries/clientpositive/alter_bucketedtable_change_column.q
##########
@@ -0,0 +1,10 @@
+--! qt:dataset:src
+create table alter_bucket_change_col_t1(key string, value string) partitioned by (ds string) clustered by (key) into 10 buckets;
+
+describe formatted alter_bucket_change_col_t1;
+
+-- Test changing name of bucket column
+
+alter table alter_bucket_change_col_t1 change key keys string;
Review comment:
added - "Serial_Num" which cover lower case, Upper case and also special char.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498439670
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
##########
@@ -3428,4 +3432,44 @@ public void testUpdatePartitionStat_doesNotUpdateStats() throws Exception {
m.invoke(hms, tbl, part, false, null);
verify(wh, never()).getFileStatusesForLocation(part.getSd().getLocation());
}
+
+ @Test(expected = InvalidOperationException.class)
Review comment:
Method is added in standalone metastore. So we need to add some unit test case in some form to validate the code the while building the standalone.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498439943
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
##########
@@ -3428,4 +3432,44 @@ public void testUpdatePartitionStat_doesNotUpdateStats() throws Exception {
m.invoke(hms, tbl, part, false, null);
verify(wh, never()).getFileStatusesForLocation(part.getSd().getLocation());
}
+
+ @Test(expected = InvalidOperationException.class)
Review comment:
Divided the test in negative and positive
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498902147
##########
File path: ql/src/test/queries/clientpositive/alter_numbuckets_partitioned_table_h23.q
##########
@@ -52,6 +52,12 @@ alter table tst1_n1 clustered by (value) into 12 buckets;
describe formatted tst1_n1;
+-- Test changing name of bucket column
+
+alter table tst1_n1 change key keys string;
+
+describe formatted tst1_n1;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498652268
##########
File path: ql/src/test/queries/clientpositive/alter_bucketedtable_change_column.q
##########
@@ -0,0 +1,10 @@
+--! qt:dataset:src
+create table alter_bucket_change_col_t1(key string, value string) partitioned by (ds string) clustered by (key) into 10 buckets;
+
+describe formatted alter_bucket_change_col_t1;
+
+-- Test changing name of bucket column
+
+alter table alter_bucket_change_col_t1 change key keys string;
Review comment:
Add test for column names with mix of upper and lower case letters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498902147
##########
File path: ql/src/test/queries/clientpositive/alter_numbuckets_partitioned_table_h23.q
##########
@@ -52,6 +52,12 @@ alter table tst1_n1 clustered by (value) into 12 buckets;
describe formatted tst1_n1;
+-- Test changing name of bucket column
+
+alter table tst1_n1 change key keys string;
+
+describe formatted tst1_n1;
Review comment:
after adding show create table test start failing. because result expect "### masked information ####". which lead to multiple test failure. The information shown by show create table is same as describe table as we are only change column name. Hence after multiple retry I decide to remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498651478
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/change/AlterTableChangeColumnOperation.java
##########
@@ -72,6 +74,10 @@ protected void doAlteration(Table table, Partition partition) throws HiveExcepti
if (desc.getNewColumnComment() != null) {
oldColumn.setComment(desc.getNewColumnComment());
}
+ if (CollectionUtils.isNotEmpty(sd.getBucketCols()) && sd.getBucketCols().contains(oldColumnName)) {
+ sd.getBucketCols().remove(oldColumnName);
+ sd.getBucketCols().add(desc.getNewColumnName());
Review comment:
Should we store it in lower-case?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/change/AlterTableChangeColumnOperation.java
##########
@@ -72,6 +74,10 @@ protected void doAlteration(Table table, Partition partition) throws HiveExcepti
if (desc.getNewColumnComment() != null) {
oldColumn.setComment(desc.getNewColumnComment());
}
+ if (CollectionUtils.isNotEmpty(sd.getBucketCols()) && sd.getBucketCols().contains(oldColumnName)) {
Review comment:
Is oldColumnName case in-sensitive?
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##########
@@ -130,6 +130,11 @@ public void alterTable(RawStore msdb, Warehouse wh, String catName, String dbnam
throw new InvalidOperationException("Invalid column " + validate);
}
+ // Validate bucketedColumns in new table
+ if (!MetaStoreServerUtils.validateBucketColumns(newt.getSd())) {
+ throw new InvalidOperationException("Bucket column doesn't match with any table columns");
Review comment:
Useful to add an error log with the column name which is missing from table columns list.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
##########
@@ -1554,4 +1555,32 @@ public static Partition createMetaPartitionObject(Table tbl, Map<String, String>
}
return tpart;
}
+
+ /**
+ * Validate bucket columns should belong to table columns.
+ * @param sd StorageDescriptor of given table
+ * @return true if bucket columns are empty or belong to table columns else false
+ */
+ public static boolean validateBucketColumns(StorageDescriptor sd) {
+ List<String> columnNames = getColumnNames(sd.getCols());
+ if(CollectionUtils.isNotEmpty(sd.getBucketCols()) && CollectionUtils.isNotEmpty(columnNames)){
+ return columnNames.containsAll(sd.getBucketCols().stream().map(String::toLowerCase).collect(Collectors.toList()));
+ } else if (CollectionUtils.isNotEmpty(sd.getBucketCols()) && CollectionUtils.isEmpty(columnNames)) {
+ return false;
+ } else {
+ return true;
+ }
+ }
+
+ /**
+ * Generate column name list from the fieldSchema list
+ * @param cols fieldSchema list
+ * @return column name list
+ */
+ public static List<String> getColumnNames(List<FieldSchema> cols) {
+ if (CollectionUtils.isNotEmpty(cols)) {
+ return cols.stream().map(FieldSchema::getName).collect(Collectors.toList());
Review comment:
Will cols always have names in lower case?
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
##########
@@ -1554,4 +1555,32 @@ public static Partition createMetaPartitionObject(Table tbl, Map<String, String>
}
return tpart;
}
+
+ /**
+ * Validate bucket columns should belong to table columns.
+ * @param sd StorageDescriptor of given table
+ * @return true if bucket columns are empty or belong to table columns else false
+ */
+ public static boolean validateBucketColumns(StorageDescriptor sd) {
+ List<String> columnNames = getColumnNames(sd.getCols());
+ if(CollectionUtils.isNotEmpty(sd.getBucketCols()) && CollectionUtils.isNotEmpty(columnNames)){
Review comment:
nit: Add space before "("
##########
File path: ql/src/test/queries/clientpositive/alter_numbuckets_partitioned_table_h23.q
##########
@@ -52,6 +52,12 @@ alter table tst1_n1 clustered by (value) into 12 buckets;
describe formatted tst1_n1;
+-- Test changing name of bucket column
+
+alter table tst1_n1 change key keys string;
+
+describe formatted tst1_n1;
Review comment:
Also check the output of "show create table" command.
##########
File path: ql/src/test/queries/clientpositive/alter_bucketedtable_change_column.q
##########
@@ -0,0 +1,10 @@
+--! qt:dataset:src
+create table alter_bucket_change_col_t1(key string, value string) partitioned by (ds string) clustered by (key) into 10 buckets;
+
+describe formatted alter_bucket_change_col_t1;
+
+-- Test changing name of bucket column
+
+alter table alter_bucket_change_col_t1 change key keys string;
Review comment:
Add test for columns names with mix of upper and lower case letters.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
##########
@@ -1554,4 +1555,32 @@ public static Partition createMetaPartitionObject(Table tbl, Map<String, String>
}
return tpart;
}
+
+ /**
+ * Validate bucket columns should belong to table columns.
+ * @param sd StorageDescriptor of given table
+ * @return true if bucket columns are empty or belong to table columns else false
+ */
+ public static boolean validateBucketColumns(StorageDescriptor sd) {
+ List<String> columnNames = getColumnNames(sd.getCols());
Review comment:
Check if CollectionUtils.isEmpty(sd.getBucketCols()) case even before getting table columns list and return true. It will avoid repeated checks below and also avoids unnecessary computes for non-bucketed tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1528: HIVE-22826: ALTER TABLE RENAME COLUMN doesn't update list of bucketed…
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1528:
URL: https://github.com/apache/hive/pull/1528#discussion_r498904232
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/change/AlterTableChangeColumnOperation.java
##########
@@ -72,6 +74,10 @@ protected void doAlteration(Table table, Partition partition) throws HiveExcepti
if (desc.getNewColumnComment() != null) {
oldColumn.setComment(desc.getNewColumnComment());
}
+ if (CollectionUtils.isNotEmpty(sd.getBucketCols()) && sd.getBucketCols().contains(oldColumnName)) {
Review comment:
As per HIVE column contract it should be case in-sensitive. But it is not handled properly in query planning of "alter table {tablename} change". So I have added toLowerCase() in query planning also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org