You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by aecc <al...@gmail.com> on 2014/02/25 16:35:46 UTC
Spark in YARN HDP problem
Hi, I'm trying to run the Spark examples in YARN and I get the following
error:
appDiagnostics: Application application_1390483691679_0124 failed 2 times
due to AM Container for appattempt_1390483691679_0124_000002 exited with
exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
This is the command to run:
SPARK_JAR=/opt/spark/assembly/target/spark-assembly_2.10-0.9.0-incubating.jar
/opt/spark/bin/spark-class org.apache.spark.deploy.yarn.Client --jar
/opt/spark/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar
--class org.apache.spark.examples.SparkPi --args yarn-standalone
And my spark-env.sh file:
export SCALA_HOME=/opt/scala
export SPARK_LIBRARY_PATH="/usr/lib/hadoop/lib/native:$SPARK_LIBRARY_PATH"
export HADOOP_CONF_DIR="/etc/hadoop/conf/"
Am I doing something wrong?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-in-YARN-HDP-problem-tp2041.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Spark in YARN HDP problem
Posted by aecc <al...@gmail.com>.
Hi Azuryy,
This is what I got:
LogType: stderr
LogLength: 85
Log Contents:
Error: Could not find or load main class
org.apache.spark.deploy.yarn.WorkerLauncher
LogType: stdout
LogLength: 0
Log Contents:
That made me thought about the consolidated jar file that I was including.
It was a simple mistake, I was including the following jar:
assembly/target/spark-assembly_2.10-0.9.0-incubating.jar
instead of:
assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar
my bad, thank you very much for the help!!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-in-YARN-HDP-problem-tp2041p2075.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Spark in YARN HDP problem
Posted by Azuryy Yu <az...@gmail.com>.
can you check the container logs and paste here?
On Feb 26, 2014 7:07 PM, "aecc" <al...@gmail.com> wrote:
> The error happens after the application is accepted:
>
> 14/02/26 12:05:08 INFO YarnClientImpl: Submitted application
> application_1390483691679_0157 to ResourceManager at XXXXXXXXX:8050
> 3483 [main] INFO
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Application
> report from ASM:
> appMasterRpcPort: 0
> appStartTime: 1393412708680
> yarnAppState: ACCEPTED
>
> 4484 [main] INFO
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Application
> report from ASM:
> appMasterRpcPort: 0
> appStartTime: 1393412708680
> yarnAppState: ACCEPTED
>
> 5485 [main] INFO
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Application
> report from ASM:
> appMasterRpcPort: 0
> appStartTime: 1393412708680
> yarnAppState: ACCEPTED
>
> 6487 [main] INFO
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Application
> report from ASM:
> appMasterRpcPort: 0
> appStartTime: 1393412708680
> yarnAppState: FAILED
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-in-YARN-HDP-problem-tp2041p2072.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: Spark in YARN HDP problem
Posted by aecc <al...@gmail.com>.
The error happens after the application is accepted:
14/02/26 12:05:08 INFO YarnClientImpl: Submitted application
application_1390483691679_0157 to ResourceManager at XXXXXXXXX:8050
3483 [main] INFO
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Application
report from ASM:
appMasterRpcPort: 0
appStartTime: 1393412708680
yarnAppState: ACCEPTED
4484 [main] INFO
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Application
report from ASM:
appMasterRpcPort: 0
appStartTime: 1393412708680
yarnAppState: ACCEPTED
5485 [main] INFO
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Application
report from ASM:
appMasterRpcPort: 0
appStartTime: 1393412708680
yarnAppState: ACCEPTED
6487 [main] INFO
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Application
report from ASM:
appMasterRpcPort: 0
appStartTime: 1393412708680
yarnAppState: FAILED
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-in-YARN-HDP-problem-tp2041p2072.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.