You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Navis Ryu <na...@nexr.com> on 2014/07/09 09:55:58 UTC

Review Request 23355: Hive unnecessarily validates table SerDes when dropping a table

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23355/
-----------------------------------------------------------

Review request for hive.


Bugs: HIVE-3392
    https://issues.apache.org/jira/browse/HIVE-3392


Repository: hive-git


Description
-------

natty@hadoop1:~$ hive
hive> add jar /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar;
Added /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar to class path
Added resource: /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar
hive> create table test (a int) row format serde 'hive.serde.JSONSerDe';            
OK
Time taken: 2.399 seconds


natty@hadoop1:~$ hive
hive> drop table test;                                                               
FAILED: Hive Internal Error: java.lang.RuntimeException(MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe hive.serde.JSONSerDe does not exist))
java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe hive.serde.JSONSerDe does not exist)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:262)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:253)
	at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:490)
	at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:943)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeDropTable(DDLSemanticAnalyzer.java:700)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:210)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:889)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe com.cloudera.hive.serde.JSONSerDe does not exist)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:211)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:260)
	... 20 more

hive> add jar /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar;
Added /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar to class path
Added resource: /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar
hive> drop table test;
OK
Time taken: 0.658 seconds
hive> 


Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4d8e10c 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 250756c 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 3a1e7fd 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 3df2690 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 2537b75 

Diff: https://reviews.apache.org/r/23355/diff/


Testing
-------


Thanks,

Navis Ryu

Re: Review Request 23355: Hive unnecessarily validates table SerDes when dropping a table

Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23355/
-----------------------------------------------------------

(Updated July 10, 2014, 6:50 a.m.)


Review request for hive.


Changes
-------

Fixed test fail & rebased to trunk.


Bugs: HIVE-3392
    https://issues.apache.org/jira/browse/HIVE-3392


Repository: hive-git


Description
-------

natty@hadoop1:~$ hive
hive> add jar /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar;
Added /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar to class path
Added resource: /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar
hive> create table test (a int) row format serde 'hive.serde.JSONSerDe';            
OK
Time taken: 2.399 seconds


natty@hadoop1:~$ hive
hive> drop table test;                                                               
FAILED: Hive Internal Error: java.lang.RuntimeException(MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe hive.serde.JSONSerDe does not exist))
java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe hive.serde.JSONSerDe does not exist)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:262)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:253)
	at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:490)
	at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:943)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeDropTable(DDLSemanticAnalyzer.java:700)
	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:210)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:889)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe com.cloudera.hive.serde.JSONSerDe does not exist)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:211)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:260)
	... 20 more

hive> add jar /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar;
Added /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar to class path
Added resource: /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar
hive> drop table test;
OK
Time taken: 0.658 seconds
hive> 


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java bbf89ef 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java fea1e47 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 3a1e7fd 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 563dbd1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 52a8096 

Diff: https://reviews.apache.org/r/23355/diff/


Testing
-------


Thanks,

Navis Ryu

Re: Review Request 23355: Hive unnecessarily validates table SerDes when dropping a table

Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23355/#review47553
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
<https://reviews.apache.org/r/23355/#comment83560>

    Can you change this message to output just the column name rather than the entire FieldSchema, to match the code that this method replaces?  String partCol = partColsIter.next().getName();


- Jason Dere


On July 9, 2014, 7:55 a.m., Navis Ryu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23355/
> -----------------------------------------------------------
> 
> (Updated July 9, 2014, 7:55 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-3392
>     https://issues.apache.org/jira/browse/HIVE-3392
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> natty@hadoop1:~$ hive
> hive> add jar /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar;
> Added /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar to class path
> Added resource: /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar
> hive> create table test (a int) row format serde 'hive.serde.JSONSerDe';            
> OK
> Time taken: 2.399 seconds
> 
> 
> natty@hadoop1:~$ hive
> hive> drop table test;                                                               
> FAILED: Hive Internal Error: java.lang.RuntimeException(MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe hive.serde.JSONSerDe does not exist))
> java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe hive.serde.JSONSerDe does not exist)
> 	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:262)
> 	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:253)
> 	at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:490)
> 	at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:943)
> 	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeDropTable(DDLSemanticAnalyzer.java:700)
> 	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:210)
> 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:889)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException SerDe com.cloudera.hive.serde.JSONSerDe does not exist)
> 	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:211)
> 	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:260)
> 	... 20 more
> 
> hive> add jar /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar;
> Added /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar to class path
> Added resource: /home/natty/source/sample-code/custom-serdes/target/custom-serdes-1.0-SNAPSHOT.jar
> hive> drop table test;
> OK
> Time taken: 0.658 seconds
> hive> 
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4d8e10c 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 250756c 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 3a1e7fd 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 3df2690 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 2537b75 
> 
> Diff: https://reviews.apache.org/r/23355/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>