You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2017/03/17 07:49:41 UTC
[jira] [Comment Edited] (SPARK-19992) spark-submit on
deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929547#comment-15929547 ]
Saisai Shao edited comment on SPARK-19992 at 3/17/17 7:48 AM:
--------------------------------------------------------------
Looks like I guess wrong from the url you provided.
If you're trying to use "spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/*", make sure the spark related jars existed under the same path in every node.
was (Author: jerryshao):
Looks like I guess wrong from the url you provided.
If you're trying to use "spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/*", make sure these jars existed in every node.
> spark-submit on deployment-mode cluster
> ---------------------------------------
>
> Key: SPARK-19992
> URL: https://issues.apache.org/jira/browse/SPARK-19992
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.0.2
> Environment: spark version 2.0.2
> hadoop version 2.6.0
> Reporter: narendra maru
>
> spark version 2.0.2
> hadoop version 2.6.0
> spark -submit command
> "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode cluster --jars /home/ec2-user/jars/hgmongonew.jar, /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar"
> after adding following in
> 1 Spark-default.conf
> spark.executor.extraJavaOptions -Dconfig.fuction.conf
> spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/*
> spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory
> spark.eventLog.enabled=true
> 2yarn-site.xml
> <property>
> <name>yarn.application.classpath</name>
> <value>
> /usr/local/hadoop-2.6.0/etc/hadoop,
> /usr/local/hadoop-2.6.0/,
> /usr/local/hadoop-2.6.0/lib/,
> /usr/local/hadoop-2.6.0/share/hadoop/common/,
> /usr/local/hadoop-2.6.0/share/hadoop/common/lib/
> /usr/local/hadoop-2.6.0/share/hadoop/hdfs/,
> /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/,
> /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/,
> /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/,
> /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/,
> /usr/local/hadoop-2.6.0/share/hadoop/yarn/,
> /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*,
> /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar
> </value>
> </property>
> Error on log:-
> Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster
> Error on terminal:-
> diagnostics: Application application_1489673977198_0002 failed 2 times due to AM Container for appattempt_1489673977198_0002_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, click on links to logs of each attempt.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org