You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by firemonk9 <dh...@gmail.com> on 2014/10/24 04:25:46 UTC
Re: Spark 1.0.0 on yarn cluster problem
Hi,
I am facing same problem. My spark-env.sh has below entries yet I see the
yarn container with only 1G and yarn only spawns two workers.
SPARK_EXECUTOR_CORES=1
SPARK_EXECUTOR_MEMORY=3G
SPARK_EXECUTOR_INSTANCES=5
Please let me know if you are able to resolve this issue.
Thank you
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-on-yarn-cluster-problem-tp7560p17175.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark 1.0.0 on yarn cluster problem
Posted by firemonk9 <dh...@gmail.com>.
Yes export worked.
Thank you
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-on-yarn-cluster-problem-tp7560p17180.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark 1.0.0 on yarn cluster problem
Posted by Andrew Or <an...@databricks.com>.
Did you `export` the environment variables? Also, are you running in client
mode or cluster mode? If it still doesn't work you can try to set these
through the spark-submit command lines --num-executors, --executor-cores,
and --executor-memory.
2014-10-23 19:25 GMT-07:00 firemonk9 <dh...@gmail.com>:
> Hi,
>
> I am facing same problem. My spark-env.sh has below entries yet I see
> the
> yarn container with only 1G and yarn only spawns two workers.
>
> SPARK_EXECUTOR_CORES=1
> SPARK_EXECUTOR_MEMORY=3G
> SPARK_EXECUTOR_INSTANCES=5
>
> Please let me know if you are able to resolve this issue.
>
> Thank you
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-on-yarn-cluster-problem-tp7560p17175.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>