You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Navis Ryu <na...@nexr.com> on 2015/02/09 03:03:52 UTC
Review Request 30780: Use session classloader instead of application
loader
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30780/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-9486
https://issues.apache.org/jira/browse/HIVE-9486
Repository: hive-git
Description
-------
PlanUtils.java use correct classloader when calling Class.forName()
?? ? Tue, 27 Jan 2015 01:08:10 -0800
Hi All,
I am not having some hive classpath issue. I think this is a bug.
I wrote my own SerDe "com.stanley.MySerde", which is a simple json serializer;
It is generally the same with the built-in SerDe
"org.apache.hadoop.hive.serde2.DelimitedJSONSerDe".
Then I issued the command:add jar /path/to/myjar.jar; (I am sure this command
worked)create table t1.json_1 row format serde "com.stanley.MySerde" location
'/user/stanley/test-data-1/' as select * from t1.plain_table;
create table t1.json_2 row format serde
"org.apache.hadoop.hive.serde2.DelimitedJSONSerDe" location
'/user/stanley/test-data-2/' as select * from t1.plain_table;
The second command will succeed but the first one will fail with
ClassNotFoundException. But if I put myjar.jar to $HIVE_HOME/lib, both command
will succeed. I went through the code of the
org.apache.hadoop.hive.ql.plan.PlanUtils.java, seems it is using
Class.forname(clzname) to load the class, I think it should use the
Thread.contextClassLoader instead, am I right?There's a similar issue here:
https://issues.apache.org/jira/browse/HIVE-6495
Here's the exception trace: java.lang.ClassNotFoundException:
com.ebay.p13n.hive.bexbat.serde.JsonLazySimpleSerDe at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:190) at
org.apache.hadoop.hive.ql.plan.PlanUtils.getTableDesc(PlanUtils.java:310) at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:5874)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8278)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8169)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9001)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9267)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427) at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980) at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:916) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
Diffs
-----
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/Utils.java 16abac2
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PrimitiveComparisonFilter.java ef459aa
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDeParameters.java ef77697
common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java 9f2f02f
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java a43520c
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java bfa8657
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatSplit.java d3d5a0f
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java 88df982
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java 8c4bca0
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/JobState.java 36b64da
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java adb50f0
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 3518edc
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HivePreWarmProcessor.java ce3b1d6
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java afe83d9
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 0ca5d22
ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveRecordReader.java ede3b6e
ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java e2ae25b
ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 9f24190
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java 0f7e833
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate/ColumnTruncateTask.java 73c6dcc
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java ff34682
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 6bef5f5
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7ac0551
ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java f75bec5
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java b62ffed
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java e26031c
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 32826e7
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java d3ad515
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java 89496ea
Diff: https://reviews.apache.org/r/30780/diff/
Testing
-------
Thanks,
Navis Ryu