You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2017/11/21 16:06:00 UTC
[jira] [Commented] (SPARK-22574) Wrong request causing Spark
Dispatcher going inactive
[ https://issues.apache.org/jira/browse/SPARK-22574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260954#comment-16260954 ]
Apache Spark commented on SPARK-22574:
--------------------------------------
User 'Gschiavon' has created a pull request for this issue:
https://github.com/apache/spark/pull/19793
> Wrong request causing Spark Dispatcher going inactive
> -----------------------------------------------------
>
> Key: SPARK-22574
> URL: https://issues.apache.org/jira/browse/SPARK-22574
> Project: Spark
> Issue Type: Bug
> Components: Mesos, Spark Submit
> Affects Versions: 2.2.0
> Reporter: German Schiavon Matteo
> Priority: Minor
> Fix For: 2.0.0, 2.1.0, 2.2.0
>
>
> When submitting a wrong _CreateSubmissionRequest_ to Spark Dispatcher is causing a bad state of Dispatcher and making it inactive as a mesos framework.
> The class CreateSubmissionRequest initialise its arguments to null as follows:
> {code:title=CreateSubmissionRequest.scala|borderStyle=solid}
> var appResource: String = null
> var mainClass: String = null
> var appArgs: Array[String] = null
> var sparkProperties: Map[String, String] = null
> var environmentVariables: Map[String, String] = null
> {code}
> There are some checks of this variable but not in all of them, for example in appArgs and environmentVariables.
> If you don't set _appArgs_ it will cause the following error:
> {code:title=error|borderStyle=solid}
> 17/11/21 14:37:24 INFO MesosClusterScheduler: Reviving Offers.
> Exception in thread "Thread-22" java.lang.NullPointerException
> at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.getDriverCommandValue(MesosClusterScheduler.scala:444)
> at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.buildDriverCommand(MesosClusterScheduler.scala:451)
> at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.org$apache$spark$scheduler$cluster$mesos$MesosClusterScheduler$$createTaskInfo(MesosClusterScheduler.scala:538)
> at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:570)
> at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:555)
> at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.scheduleTasks(MesosClusterScheduler.scala:555)
> at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.resourceOffers(MesosClusterScheduler.scala:621)
> {code}
> Because it's trying to access to it without checking whether is null or not.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org