You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ajatix <aj...@sigmoidanalytics.com> on 2014/06/26 23:01:41 UTC

Spark-submit failing on cluster

My setup ---
I have a private cluster running on 4 nodes. I want to use the spark-submit
script to execute spark applications on the cluster. I am using Mesos to
manage the cluster. 

This is the command I ran on local mode, which ran successfully ---
 ./bin/spark-submit --master local --class org.apache.spark.examples.SparkPi
/opt/spark-examples-1.0.0-hadoop2.4.0.jar 100

This is the command I ran on cluster mode, which did failed to run ---
./bin/spark-submit --master mesos://<ip-addr>:5050 --class
org.apache.spark.examples.SparkPi /opt/spark-examples-1.0.0-hadoop2.4.0.jar
100

(storing jar on hdfs mode)
./bin/spark-submit --master mesos://<ip-addr>:5050 --class
org.apache.spark.examples.SparkPi
hdfs://<ip-addr>/jars/spark-examples-1.0.0-hadoop2.4.0.jar 100

This is the stderr displayed in mesos sandbox --- 
sh: /home/<user>/spark-1.0.0/sbin/spark-executor: No such file or directory

My SPARK_EXECUTOR_URI location is at
'hdfs://<ip-addr>:9000/new/spark-1.0.0-hadoop-2.4.0.tgz'. How do I give the
parameter to spark-submit to use the executor location. Does spark-submit
not refer to spark-env.sh file? because that is what spark-shell uses and
runs smoothly.

Tell me if my command is incorrect or where I'm going wrong with the usage.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-failing-on-cluster-tp8369.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark-submit failing on cluster

Posted by ajatix <aj...@sigmoidanalytics.com>.
Rectified the issue by providing the executor uri location in the input

./bin/spark-submit --master mesos://<ip-addr>:5050 --class
org.apache.spark.examples.SparkPi --driver-java-options
-Dspark.executor.uri=hdfs://<ip-addr>:9000/new/spark-1.0.0-hadoop-2.4.0.tgz
/opt/spark-examples-1.0.0-hadoop2.4.0.jar 10

I am still not able to give the path of the jar in hdfs. This only seems to
work for locally stored jars.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-failing-on-cluster-tp8369p8378.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.