Running Spark Job on Yarn from Java Code

[Rahul Swaminathan <ra...@duke.edu>]

Hi all,

I am trying to run a simple Spark_Pi application through Yarn from Java code. I have the Spark_Pi class and everything works fine if I run on Spark. However, when I set master to "yarn-client" and set yarn mode to true, I keep getting exceptions. I suspect this has something to do with the wrong jars being in my class path, but I can't seem to find any documentation on exactly what I need. Can someone post an example of running a Spark Job from Java code on Yarn? What config properties and jars were necessary?

Thanks!
Rahul

Re: Running Spark Job on Yarn from Java Code

[Kyle Lin <ky...@gmail.com>]

Hi there

I also got exception when running PI example on YARN

Spark version: spark-1.1.1-bin-hadoop2.4

My environment: Hortonworks HDP 2.2

My command:
./bin/spark-submit --master yarn-cluster --class
org.apache.spark.examples.SparkPi lib/spark-examples*.jar 10

Output logs:
14/12/17 14:06:32 INFO yarn.Client: Application report from ResourceManager:
 application identifier: application_1418282263779_0024
 appId: 24
 clientToAMToken: null
 appDiagnostics: Application application_1418282263779_0024 failed 2 times
due to AM Container for appattempt_1418282263779_0024_000002 exited with
 exitCode: 1
For more detailed output, check application tracking page:
http://sparkvm.localdomain:8088/proxy/application_1418282263779_0024/Then,
click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1418282263779_0024_02_000001
Exit code: 1
Exception message:
/hadoop/yarn/local/usercache/root/appcache/application_1418282263779_0024/container_1418282263779_0024_02_000001/launch_container.sh:
line 27:
$PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/__app__.jar:$PWD/*:
bad substitution

Stack trace: ExitCodeException exitCode=1:
/hadoop/yarn/local/usercache/root/appcache/application_1418282263779_0024/container_1418282263779_0024_02_000001/launch_container.sh:
line 27:
$PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/__app__.jar:$PWD/*:
bad substitution

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
 appMasterHost: N/A
 appQueue: default
 appMasterRpcPort: -1
 appStartTime: 1418796384166
 yarnAppState: FAILED
 distributedFinalState: FAILED
 appTrackingUrl:
http://sparkvm.localdomain:8088/cluster/app/application_1418282263779_0024
 appUser: root
Exception in thread "main" org.apache.spark.SparkException: Application is
not successful
at org.apache.spark.deploy.yarn.Client.run(Client.scala:98)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:176)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Kyle


2014-12-17 2:45 GMT+08:00 Rahul Swaminathan <ra...@duke.edu>:
>
>  Hi all,
>
>  I am trying to run a simple Spark_Pi application through Yarn from Java
> code. I have the Spark_Pi class and everything works fine if I run on
> Spark. However, when I set master to “yarn-client” and set yarn mode to
> true, I keep getting exceptions. I suspect this has something to do with
> the wrong jars being in my class path, but I can’t seem to find any
> documentation on exactly what I need. Can someone post an example of
> running a Spark Job from Java code on Yarn? What config properties and jars
> were necessary?
>
>  Thanks!
> Rahul
>