You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Rui Li (JIRA)" <ji...@apache.org> on 2014/12/18 12:48:13 UTC
[jira] [Commented] (HIVE-8722) Enhance InputSplitShims to extend
InputSplitWithLocationInfo [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14251557#comment-14251557 ]
Rui Li commented on HIVE-8722:
------------------------------
I got this exception which also seems related:
{noformat}
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - 14/12/18 12:25:18 DEBUG rdd.HadoopRDD: SplitLocationInfo and other new Hadoop classes are unavailable. Using the older Hadoop location info code.
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - java.lang.ClassNotFoundException: org.apache.hadoop.mapred.InputSplitWithLocationInfo
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.security.AccessController.doPrivileged(Native Method)
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.lang.Class.forName0(Native Method)
2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.lang.Class.forName(Class.java:190)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD$SplitInfoReflections.<init>(HadoopRDD.scala:381)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD$.liftedTree1$1(HadoopRDD.scala:391)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD$.<init>(HadoopRDD.scala:390)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD$.<clinit>(HadoopRDD.scala)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:179)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:197)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:206)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.Option.getOrElse(Option.scala:120)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD.partitions(RDD.scala:204)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:206)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.Option.getOrElse(Option.scala:120)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD.partitions(RDD.scala:204)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:79)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:80)
2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:193)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:191)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.Option.getOrElse(Option.scala:120)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD.dependencies(RDD.scala:191)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGScheduler.visit$1(DAGScheduler.scala:301)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGScheduler.getParentStages(DAGScheduler.scala:313)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGScheduler.newStage(DAGScheduler.scala:247)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:734)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1389)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.actor.ActorCell.invoke(ActorCell.scala:487)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.dispatch.Mailbox.run(Mailbox.scala:220)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
2014-12-18 12:25:18,402 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{noformat}
> Enhance InputSplitShims to extend InputSplitWithLocationInfo [Spark Branch]
> ---------------------------------------------------------------------------
>
> Key: HIVE-8722
> URL: https://issues.apache.org/jira/browse/HIVE-8722
> Project: Hive
> Issue Type: Improvement
> Reporter: Jimmy Xiang
>
> We got thie following exception in hive.log:
> {noformat}
> 2014-11-03 11:45:49,865 DEBUG rdd.HadoopRDD
> (Logging.scala:logDebug(84)) - Failed to use InputSplitWithLocations.
> java.lang.ClassCastException: Cannot cast
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit
> to org.apache.hadoop.mapred.InputSplitWithLocationInfo
> at java.lang.Class.cast(Class.java:3094)
> at org.apache.spark.rdd.HadoopRDD.getPreferredLocations(HadoopRDD.scala:278)
> at org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:216)
> at org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:216)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.preferredLocations(RDD.scala:215)
> at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:1303)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$2.apply$mcVI$sp(DAGScheduler.scala:1313)
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$2.apply(DAGScheduler.scala:1312)
> {noformat}
> My understanding is that the split location info helps Spark to execute tasks more efficiently. This could help other execution engine too. So we should consider to enhance InputSplitShim to implement InputSplitWithLocationInfo if possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)