You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Rui Li (JIRA)" <ji...@apache.org> on 2017/12/20 05:16:03 UTC
[jira] [Commented] (HIVE-18289) Fix jar dependency when enable rdd
cache in Hive on Spark
[ https://issues.apache.org/jira/browse/HIVE-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297893#comment-16297893 ]
Rui Li commented on HIVE-18289:
-------------------------------
It seems the reason is OrcStruct doesn't have an empty constructor.
[~owen.omalley], any thoughts on this? Thanks.
> Fix jar dependency when enable rdd cache in Hive on Spark
> ---------------------------------------------------------
>
> Key: HIVE-18289
> URL: https://issues.apache.org/jira/browse/HIVE-18289
> Project: Hive
> Issue Type: Bug
> Reporter: liyunzhang
> Assignee: liyunzhang
>
> running DS/query28 when enabling HIVE-17486's 4th patch
> on tpcds_bin_partitioned_orc_10 whether on spark local or yarn mode
> command
> {code}
> set spark.local=yarn-client;
> echo 'use tpcds_bin_partitioned_orc_10;source query28.sql;'|hive --hiveconf spark.app.name=query28.sql --hiveconf hive.spark.optimize.shared.work=true -i testbench.settings -i query28.sql.setting
> {code}
> the exception
> {code}
> ava.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.io.orc.OrcStruct.<init>()
> 748678 at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134) ~[hadoop-common-2.7.3.jar:?]
> 748679 at org.apache.hadoop.io.WritableUtils.clone(WritableUtils.java:217) ~[hadoop-common-2.7.3.jar:?]
> 748680 at org.apache.hadoop.hive.ql.exec.spark.MapInput$CopyFunction.call(MapInput.java:85) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0. 0-SNAPSHOT]
> 748681 at org.apache.hadoop.hive.ql.exec.spark.MapInput$CopyFunction.call(MapInput.java:72) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0. 0-SNAPSHOT]
> 748682 at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1031) ~[spark-core_2.11-2. 0.0.jar:2.0.0]
> 748683 at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1031) ~[spark-core_2.11-2. 0.0.jar:2.0.0]
> 748684 at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) ~[scala-library-2.11.8.jar:?]
> 748685 at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:214) ~[spark-core_2.11-2.0.0.jar:2. 0.0]
> 748686 at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:919) ~[spark-core_2.11-2.0.0. jar:2.0.0]
> 748687 at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:910) ~[spark-core_2.11-2.0.0. jar:2.0.0]
> 748688 at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866) ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748689 at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:910) ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748690 at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:668) ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748691 at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330) ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748692 at org.apache.spark.rdd.RDD.iterator(RDD.scala:281) ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748693 at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748694 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748695 at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) ~[spark-core_2.11-2.0.0.jar:2.0.0]
> 748696 at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) ~[spark-core_2.11-2
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)