You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Rodrick Brown <ro...@orchard-app.com> on 2016/02/12 21:16:50 UTC
Unable to start more than two executors on Mesos with Spark
I'm having an issue with Spark when running on Mesos in coarse grain mode no
matter what options I use or how many free resources I have I only ever see
two executors start.
If I run Spark in fine grain mode on Mesos I see tasks eventually ramp up to
the desired number I specified.
Anyone know what could be causing this?
I'm starting my job using the following commands --
/usr/bin/timeout 3600 /opt/spark-1.5.1/bin/spark-submit --master "mesos://zk
://prod-zookeeper-1.aws.xxx.com:2181,prod-zookeeper-2.aws.xxx.com:2181,prod-
zookeeper-3.aws.xxx.com:2181/mesos" --conf spark.ui.port=31278 --conf
spark.mesos.coarse=true --conf spark.mesos.extra.cores=3 --conf
spark.cores.max=5 --conf spark.mesos.constraints="rack:spark" --conf
spark.sql.tungsten.enabled=true --class
com.orchard.dataloader.library.originators.prosper.LoadTrade_Prosper --total-
executor-cores 3 --driver-memory 1536M --executor-memory 3011M --jars
/data/orchard/etc/load-tradedata-accumulo-prod /data/orchard/jars/dataloader-
library-59e0441cd06544442d2a84f5842b701e81097988-assembled.jar
I'm using spark-1.5.1 and Mesos 0.26 Chronos-2.4.0 is starting the spark
driver.
If anyone have any ideas on what/where I should be focusing on to debug this?
\--
**Rodrick Brown** / Systems Engineer
+1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)
**Orchard Platform**
101 5th Avenue, 4th Floor, New York, NY 10003
[http://www.orchardplatform.com](http://www.orchardplatform.com/)
[Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.
Re: Unable to start more than two executors on Mesos with Spark
Posted by haosdent <ha...@gmail.com>.
Hi, I think in coarse grain mode, Spark would schedule and allocate
resource by itself, just like what it do in YARN. So you only could see two
executors.
I think you could get more details from
http://spark.apache.org/docs/latest/running-on-mesos.html#mesos-run-modes
On Sat, Feb 13, 2016 at 4:16 AM, Rodrick Brown <ro...@orchard-app.com>
wrote:
> I'm having an issue with Spark when running on Mesos in coarse grain mode
> no matter what options I use or how many free resources I have I only ever
> see two executors start.
>
> If I run Spark in fine grain mode on Mesos I see tasks eventually ramp up
> to the desired number I specified.
> Anyone know what could be causing this?
>
> I'm starting my job using the following commands --
>
> /usr/bin/timeout 3600 /opt/spark-1.5.1/bin/spark-submit --master "
> mesos://zk://prod-zookeeper-1.aws.xxx.com:2181,prod-zookeeper-2.aws.xxx.com:2181,prod-zookeeper-3.aws.xxx.com:2181/mesos"
> --conf spark.ui.port=31278 --conf spark.mesos.coarse=true --conf
> spark.mesos.extra.cores=3 --conf spark.cores.max=5 --conf
> spark.mesos.constraints="rack:spark" --conf spark.sql.tungsten.enabled=true
> --class
> com.orchard.dataloader.library.originators.prosper.LoadTrade_Prosper
> --total-executor-cores 3 --driver-memory 1536M --executor-memory 3011M
> --jars /data/orchard/etc/load-tradedata-accumulo-prod
> /data/orchard/jars/dataloader-library-59e0441cd06544442d2a84f5842b701e81097988-assembled.jar
>
>
> I'm using spark-1.5.1 and Mesos 0.26 Chronos-2.4.0 is starting the spark
> driver.
>
>
> If anyone have any ideas on what/where I should be focusing on to debug
> this?
>
> --
>
> *Rodrick Brown* / Systems Engineer
>
> +1 917 445 6839 / rodrick@orchardplatform.com
> <ch...@orchardplatform.com>
>
> *Orchard Platform*
>
> 101 5th Avenue, 4th Floor, New York, NY 10003
>
> http://www.orchardplatform.com
>
> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending
> Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>
> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
> for the use of the addressee only. If you are not an intended recipient of
> this communication, please delete it immediately and notify the sender by
> return email. Unauthorized reading, dissemination, distribution or copying
> of this communication is prohibited. This communication does not constitute
> an offer to sell or a solicitation of an indication of interest to purchase
> any loan, security or any other financial product or instrument, nor is it
> an offer to sell or a solicitation of an indication of interest to purchase
> any products or services to any persons who are prohibited from receiving
> such information under applicable law. The contents of this communication
> may not be accurate or complete and are subject to change without notice.
> As such, Orchard App, Inc. (including its subsidiaries and affiliates,
> "Orchard") makes no representation regarding the accuracy or completeness
> of the information contained herein. The intended recipient is advised to
> consult its own professional advisors, including those specializing in
> legal, tax and accounting matters. Orchard does not provide legal, tax or
> accounting advice.
>
--
Best Regards,
Haosdent Huang