You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by SparkUser6 <al...@gmail.com> on 2018/05/08 09:14:55 UTC
Error submitting Spark Job in yarn-cluster mode on EMR
I have a simple program that works fine in the local mode. But I am having
issues when I try to run the program in yarn-cluster mode. I know usually
no such method happens when compile and run version mismatch but I made sure
I took the same version.
205 [main] INFO org.spark_project.jetty.server.ServerConnector - Started
Spark@29539e36{HTTP/1.1}{0.0.0.0:4040}
205 [main] INFO org.spark_project.jetty.server.Server - Started @3265ms
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.spark.internal.config.package$.APP_CALLER_CONTEXT()Lorg/apache/spark/internal/config/OptionalConfigEntry;
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:163)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at
com.voicebase.etl.PhoenixToElasticSearch.main(PhoenixToElasticSearch.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Error submitting Spark Job in yarn-cluster mode on EMR
Posted by Marco Mistroni <mm...@gmail.com>.
Did you by any chances left a sparkSession.setMaster("local") lurking in
your code?
Last time i checked, to run on yarn you have to package a 'fat jar'. could
you make sure the spark depedencies in your jar matches the version you are
running on Yarn?
alternatively please share code including how you submit your application
to spark
FYI this is the command i am using to submit a program to spark
spark-submit --master yarn --deploy-mode cluster --class <MyTaskClass>
<myjar.jar> <input args>
hth
On Tue, May 8, 2018 at 10:14 AM, SparkUser6 <al...@gmail.com>
wrote:
> I have a simple program that works fine in the local mode. But I am having
> issues when I try to run the program in yarn-cluster mode. I know usually
> no such method happens when compile and run version mismatch but I made
> sure
> I took the same version.
>
> 205 [main] INFO org.spark_project.jetty.server.ServerConnector -
> Started
> Spark@29539e36{HTTP/1.1}{0.0.0.0:4040}
> 205 [main] INFO org.spark_project.jetty.server.Server - Started @3265ms
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.spark.internal.config.package$.APP_CALLER_
> CONTEXT()Lorg/apache/spark/internal/config/OptionalConfigEntry;
> at org.apache.spark.deploy.yarn.Client.submitApplication(
> Client.scala:163)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(
> YarnClientSchedulerBackend.scala:56)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(
> TaskSchedulerImpl.scala:156)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
> at
> org.apache.spark.api.java.JavaSparkContext.<init>(
> JavaSparkContext.scala:58)
> at
> com.voicebase.etl.PhoenixToElasticSearch.main(PhoenixToElasticSearch.java:
> 54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>