You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by drewrobb <dr...@gmail.com> on 2016/10/13 17:42:04 UTC
No way to set mesos cluster driver memory overhead?
When using spark on mesos and deploying a job in cluster mode using
dispatcher, there appears to be no memory overhead configuration for the
launched driver processes ("--driver-memory" is the same as Xmx which is the
same as the memory quota). This makes it almost a guarantee that a long
running driver will be OOM killed by mesos. Yarn cluster mode has an
equivalent option -- spark.yarn.driver.memoryOverhead. Is there some way to
configure driver memory overhead that I'm missing?
Bigger picture question-- Is it even best practice to deploy long running
spark streaming jobs using dispatcher? I could alternatively launch the
driver by itself using marathon for example, where it would be trivial to
grant the process additional memory.
Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/No-way-to-set-mesos-cluster-driver-memory-overhead-tp27897.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: No way to set mesos cluster driver memory overhead?
Posted by Rodrick Brown <ro...@orchardplatform.com>.
On Thu, Oct 13, 2016 at 1:42 PM, drewrobb <dr...@gmail.com> wrote:
> When using spark on mesos and deploying a job in cluster mode using
> dispatcher, there appears to be no memory overhead configuration for the
> launched driver processes ("--driver-memory" is the same as Xmx which is
> the
> same as the memory quota). This makes it almost a guarantee that a long
> running driver will be OOM killed by mesos. Yarn cluster mode has an
> equivalent option -- spark.yarn.driver.memoryOverhead. Is there some way
> to
> configure driver memory overhead that I'm missing?
>
> Bigger picture question-- Is it even best practice to deploy long running
> spark streaming jobs using dispatcher? I could alternatively launch the
> driver by itself using marathon for example, where it would be trivial to
> grant the process additional memory.
>
> Thanks!
>
> I wrote a simple wrapper to provide a little head room, I'm currently
using the Chronos framework to run my spark jobs which provides some handy
environment variables like CHRONOS_RESOURCE_MEM which is what's offered
from Mesos.
execString.append('--driver-memory {:.0f}M'.format(.7 *
float(os.environ.get('CHRONOS_RESOURCE_MEM',params.get('memory')))))
execString.append('--executor-memory {:.0f}M'.format(.98 *
float(os.environ.get('CHRONOS_RESOURCE_MEM',params.get('memory')))))
execString.append('--jars {} {} {}'.format(params.get('classpath'),
params.get('jarfile'), params.get('optargs')))
i.e. SPARK_MESOS_DRIVER_MEM = CHRONOS_RESOURCE_MEM * .7
We basically set the upper limit for the driver to use 70% of what's
allocated in total to Mesos.
This will help get around Mesos killing long running jobs because of OOM.
--
[image: Orchard Platform] <http://www.orchardplatform.com/>
*Rodrick Brown */ *SRE - DevOPs*
9174456839 / rodrick@orchardplatform.com
Orchard Platform
101 5th Avenue, 4th Floor, New York, NY
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.
Re: No way to set mesos cluster driver memory overhead?
Posted by Michael Gummelt <mg...@mesosphere.io>.
We see users run both in the dispatcher and marathon. I generally prefer
marathon, because there's a higher likelihood it's going to have some
feature you need that the dispatcher lacks (like in this case).
It doesn't look like we support overhead for the driver.
On Thu, Oct 13, 2016 at 10:42 AM, drewrobb <dr...@gmail.com> wrote:
> When using spark on mesos and deploying a job in cluster mode using
> dispatcher, there appears to be no memory overhead configuration for the
> launched driver processes ("--driver-memory" is the same as Xmx which is
> the
> same as the memory quota). This makes it almost a guarantee that a long
> running driver will be OOM killed by mesos. Yarn cluster mode has an
> equivalent option -- spark.yarn.driver.memoryOverhead. Is there some way
> to
> configure driver memory overhead that I'm missing?
>
> Bigger picture question-- Is it even best practice to deploy long running
> spark streaming jobs using dispatcher? I could alternatively launch the
> driver by itself using marathon for example, where it would be trivial to
> grant the process additional memory.
>
> Thanks!
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/No-way-to-set-mesos-cluster-driver-
> memory-overhead-tp27897.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
--
Michael Gummelt
Software Engineer
Mesosphere
Re: No way to set mesos cluster driver memory overhead?
Posted by drewrobb <dr...@gmail.com>.
It seems like this is a real issue, so I've opened an issue:
https://issues.apache.org/jira/browse/SPARK-17928
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/No-way-to-set-mesos-cluster-driver-memory-overhead-tp27897p27901.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org