You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by syepes <sy...@gmail.com> on 2015/12/17 21:03:16 UTC
Spark 1.6 - YARN Cluster Mode
Hello,
This week I have been testing 1.6 (#d509194b) in our HDP 2.3 platform and
its been working pretty ok, at the exception of the YARN cluster deployment
mode.
Note that with 1.5 using the same "spark-props.conf" and "spark-env.sh"
config files the cluster mode works as expected.
Has anyone else also tried the cluster mode in 1.6?
Problem reproduction:
----
# spark-submit --master yarn --deploy-mode cluster --num-executors 1
--properties-file $PWD/spark-props.conf --class
org.apache.spark.examples.SparkPi
/opt/spark/lib/spark-examples-1.6.0-SNAPSHOT-hadoop2.7.1.jar
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ApplicationMaster
spark-props.conf
-----------------------------
spark.driver.extraJavaOptions -Dhdp.version=2.3.2.0-2950
spark.driver.extraLibraryPath
/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
spark.executor.extraJavaOptions -Dhdp.version=2.3.2.0-2950
spark.executor.extraLibraryPath
/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
-----------------------------
I will try to do some more debugging on this issue.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-YARN-Cluster-Mode-tp25729.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark 1.6 - YARN Cluster Mode
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Try adding these properties:
spark.driver.extraJavaOptions -Dhdp.version=2.3.2.0-2950
spark.yarn.am.extraJavaOptions -Dhdp.version=2.3.2.0-2950
There was a similar discussion with Spark 1.3.0 over here
http://stackoverflow.com/questions/29470542/spark-1-3-0-running-pi-example-on-yarn-fails
Thanks
Best Regards
On Fri, Dec 18, 2015 at 1:33 AM, syepes <sy...@gmail.com> wrote:
> Hello,
>
> This week I have been testing 1.6 (#d509194b) in our HDP 2.3 platform and
> its been working pretty ok, at the exception of the YARN cluster deployment
> mode.
> Note that with 1.5 using the same "spark-props.conf" and "spark-env.sh"
> config files the cluster mode works as expected.
>
> Has anyone else also tried the cluster mode in 1.6?
>
>
> Problem reproduction:
> ----
> # spark-submit --master yarn --deploy-mode cluster --num-executors 1
> --properties-file $PWD/spark-props.conf --class
> org.apache.spark.examples.SparkPi
> /opt/spark/lib/spark-examples-1.6.0-SNAPSHOT-hadoop2.7.1.jar
>
> Error: Could not find or load main class
> org.apache.spark.deploy.yarn.ApplicationMaster
>
> spark-props.conf
> -----------------------------
> spark.driver.extraJavaOptions -Dhdp.version=2.3.2.0-2950
> spark.driver.extraLibraryPath
>
> /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
> spark.executor.extraJavaOptions -Dhdp.version=2.3.2.0-2950
> spark.executor.extraLibraryPath
>
> /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
> -----------------------------
>
> I will try to do some more debugging on this issue.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-YARN-Cluster-Mode-tp25729.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>