You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Phabricator (JIRA)" <ji...@apache.org> on 2013/04/25 07:22:17 UTC
[jira] [Updated] (HIVE-4409) Prevent incompatible column type
changes
[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-4409:
------------------------------
Attachment: HIVE-4409.D10539.1.patch
dilipj requested code review of "HIVE-4409 [jira] Prevent incompatible column type changes".
Reviewers: JIRA
Disable incompatible column type changes
JIRA: HIVE-4409
If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter.
Example:
CREATE TABLE test_table123 (a INT, b MAP<STRING, STRING>) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE;
INSERT OVERWRITE TABLE test_table123 PARTITION(ds="foo1") SELECT 1, MAP("a1", "b1") FROM src LIMIT 1;
SELECT * from test_table123 WHERE ds="foo1";
SET hive.metastore.disallow.invalid.col.type.changes=true;
ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING);
SELECT * from test_table123 WHERE ds="foo1";
The last SELECT fails with the following exception:
Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108)
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D10539
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
ql/src/test/queries/clientnegative/disallow_incompatible_type_change_on1.q
ql/src/test/queries/clientnegative/disallow_incompatible_type_change_on2.q
ql/src/test/queries/clientpositive/disallow_incompatible_type_change_off.q
ql/src/test/results/clientnegative/disallow_incompatible_type_change_on1.q.out
ql/src/test/results/clientnegative/disallow_incompatible_type_change_on2.q.out
ql/src/test/results/clientpositive/disallow_incompatible_type_change_off.q.out
MANAGE HERALD RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/25215/
To: JIRA, dilipj
> Prevent incompatible column type changes
> ----------------------------------------
>
> Key: HIVE-4409
> URL: https://issues.apache.org/jira/browse/HIVE-4409
> Project: Hive
> Issue Type: Improvement
> Components: CLI, Metastore
> Affects Versions: 0.10.0
> Reporter: Dilip Joseph
> Assignee: Dilip Joseph
> Priority: Minor
> Attachments: HIVE-4409.D10539.1.patch
>
>
> If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter.
> Example:
> CREATE TABLE test_table123 (a INT, b MAP<STRING, STRING>) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE;
> INSERT OVERWRITE TABLE test_table123 PARTITION(ds="foo1") SELECT 1, MAP("a1", "b1") FROM src LIMIT 1;
> SELECT * from test_table123 WHERE ds="foo1";
> SET hive.metastore.disallow.invalid.col.type.changes=true;
> ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING);
> SELECT * from test_table123 WHERE ds="foo1";
> The last SELECT fails with the following exception:
> Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
> at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
> at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira