You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/04/26 15:05:00 UTC
[jira] [Assigned] (SPARK-23830) Spark on YARN in cluster deploy
mode fail with NullPointerException when a Spark application is a Scala
class not object
[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-23830:
------------------------------------
Assignee: (was: Apache Spark)
> Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-23830
> URL: https://issues.apache.org/jira/browse/SPARK-23830
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.3.0
> Reporter: Jacek Laskowski
> Priority: Trivial
>
> As reported on StackOverflow in [Why does Spark on YARN fail with “Exception in thread ”Driver“ java.lang.NullPointerException”?|https://stackoverflow.com/q/49564334/1305344] the following Spark application fails with {{Exception in thread "Driver" java.lang.NullPointerException}} with Spark on YARN in cluster deploy mode:
> {code}
> class MyClass {
> def main(args: Array[String]): Unit = {
> val c = new MyClass()
> c.process()
> }
> def process(): Unit = {
> val sparkConf = new SparkConf().setAppName("my-test")
> val sparkSession: SparkSession = SparkSession.builder().config(sparkConf).getOrCreate()
> import sparkSession.implicits._
> ....
> }
> ...
> }
> {code}
> The exception is as follows:
> {code}
> 18/03/29 20:07:52 INFO ApplicationMaster: Starting the user application in a separate Thread
> 18/03/29 20:07:52 INFO ApplicationMaster: Waiting for spark context initialization...
> Exception in thread "Driver" java.lang.NullPointerException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
> {code}
> I think the reason for the exception {{Exception in thread "Driver" java.lang.NullPointerException}} is due to [the following code|https://github.com/apache/spark/blob/v2.3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L700-L701]:
> {code}
> val mainMethod = userClassLoader.loadClass(args.userClass)
> .getMethod("main", classOf[Array[String]])
> {code}
> So when {{mainMethod}} is used in [the following code|https://github.com/apache/spark/blob/v2.3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L706] it simply gives NPE.
> {code}
> mainMethod.invoke(null, userArgs.toArray)
> {code}
> That could be easily avoided with an extra check if the {{mainMethod}} is initialized and give a user a message what may have been a reason.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org