You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2016/06/01 21:18:40 UTC
--driver-cores for Standalone and YARN only?! What about Mesos?
Hi,
I'm reviewing the code of spark-submit and can see that although
--driver-cores is said to be for Standalone and YARN only, it is
applicable for Mesos [1].
➜ spark git:(master) ✗ ./bin/spark-shell --help
Usage: ./bin/spark-shell [options]
...
Spark standalone with cluster deploy mode only:
--driver-cores NUM Cores for driver (Default: 1).
...
YARN-only:
--driver-cores NUM Number of cores used by the driver, only
in cluster mode
(Default: 1).
I think Mesos has been overlooked (as it's not even included in the
--help). I also can't find that the default number of cores for the
driver for the option is 1.
I can see few things to fix:
1. Have --driver-cores in the "main" help with no separation for
standalone and YARN.
2. Add note that it works only for cluster deploy mode.
3. Remove (Default: 1)
Please confirm (or fix) my understanding before I file a JIRA issue. Thanks!
[1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L475-L476
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: --driver-cores for Standalone and YARN only?! What about Mesos?
Posted by Holden Karau <ho...@pigscanfly.ca>.
Also seems like this might be better suited for dev@
On Thursday, June 2, 2016, Sun Rui <su...@163.com> wrote:
> yes, I think you can fire a JIRA issue for this.
> But why removing the default value. Seems the default core is 1 according
> to
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/rest/mesos/MesosRestServer.scala#L110
>
> On Jun 2, 2016, at 05:18, Jacek Laskowski <jacek@japila.pl
> <javascript:_e(%7B%7D,'cvml','jacek@japila.pl');>> wrote:
>
> Hi,
>
> I'm reviewing the code of spark-submit and can see that although
> --driver-cores is said to be for Standalone and YARN only, it is
> applicable for Mesos [1].
>
> ➜ spark git:(master) ✗ ./bin/spark-shell --help
> Usage: ./bin/spark-shell [options]
> ...
> Spark standalone with cluster deploy mode only:
> --driver-cores NUM Cores for driver (Default: 1).
> ...
> YARN-only:
> --driver-cores NUM Number of cores used by the driver, only
> in cluster mode
> (Default: 1).
>
> I think Mesos has been overlooked (as it's not even included in the
> --help). I also can't find that the default number of cores for the
> driver for the option is 1.
>
> I can see few things to fix:
>
> 1. Have --driver-cores in the "main" help with no separation for
> standalone and YARN.
> 2. Add note that it works only for cluster deploy mode.
> 3. Remove (Default: 1)
>
> Please confirm (or fix) my understanding before I file a JIRA issue.
> Thanks!
>
> [1]
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L475-L476
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <javascript:_e(%7B%7D,'cvml','user-unsubscribe@spark.apache.org');>
> For additional commands, e-mail: user-help@spark.apache.org
> <javascript:_e(%7B%7D,'cvml','user-help@spark.apache.org');>
>
>
>
>
>
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Re: --driver-cores for Standalone and YARN only?! What about Mesos?
Posted by Sun Rui <su...@163.com>.
yes, I think you can fire a JIRA issue for this.
But why removing the default value. Seems the default core is 1 according to https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/rest/mesos/MesosRestServer.scala#L110
On Jun 2, 2016, at 05:18, Jacek Laskowski <ja...@japila.pl> wrote:
Hi,
I'm reviewing the code of spark-submit and can see that although
--driver-cores is said to be for Standalone and YARN only, it is
applicable for Mesos [1].
➜ spark git:(master) ✗ ./bin/spark-shell --help
Usage: ./bin/spark-shell [options]
...
Spark standalone with cluster deploy mode only:
--driver-cores NUM Cores for driver (Default: 1).
...
YARN-only:
--driver-cores NUM Number of cores used by the driver, only
in cluster mode
(Default: 1).
I think Mesos has been overlooked (as it's not even included in the
--help). I also can't find that the default number of cores for the
driver for the option is 1.
I can see few things to fix:
1. Have --driver-cores in the "main" help with no separation for
standalone and YARN.
2. Add note that it works only for cluster deploy mode.
3. Remove (Default: 1)
Please confirm (or fix) my understanding before I file a JIRA issue. Thanks!
[1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L475-L476
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org