You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by mkhaitman <ma...@chango.com> on 2015/10/23 21:05:55 UTC
Spark.Executor.Cores question
Regarding the 'spark.executor.cores' config option in a Standalone spark
environment, I'm curious about whether there's a way to enforce the
following logic:
*- Max cores per executor = 4*
** Max executors PER application PER worker = 1*
In order to force better balance across all workers, I want to ensure that a
single spark job can only ever use a specific upper limit on the number of
cores for each executor it holds, however, do not want a situation where it
can spawn 3 executors on a worker and only 1/2 on the others. Some spark
jobs end up using much more memory during aggregation tasks (joins /
groupBy's) which is more heavily impacted by the number of cores per
executor for that job.
If this kind of setup/configuration doesn't already exist for Spark, and
others see the benefit of what I mean by this, where would be the best
location to insert this logic?
Mark.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Executor-Cores-question-tp14763.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark.Executor.Cores question
Posted by mkhaitman <ma...@chango.com>.
Unfortunately setting the executor memory to prevent multiple executors from
the same framework would inherently mean that we'd need to set just over
half the available worker memory for each node. So if each node had 32GB of
worker memory, then the application would need to set 17GB to absolutely
ensure that it never gets 2 executors on the same host (but that's way too
much memory to request per node in our case haha).
I'll look into adding this type of config option myself when I get some
time, since I think it would still be valuable to prevent a memory-intensive
application from taking down a bunch of spark workers just because their
executors are going bananas :P
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Executor-Cores-question-tp14763p14826.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark.Executor.Cores question
Posted by Richard Marscher <rm...@localytics.com>.
Ah I see, that's a bit more complicated =). If it's possible, would using
`spark.executor.memory` to set the available worker memory used by
executors help alleviate the problem of running on a node that already has
an executor on it? I would assume that would have a constant worst case
overhead per worker and shouldn't matter if two executors on the same node
are from one application or two applications. Maybe you have data that
states otherwise? I suppose it depends on what resources are causing
problems when the executors are imbalanced across the cluster. That could
be an indication of possibly not leaving enough free RAM outside the worker
and executors heap allocations on worker nodes.
On Tue, Oct 27, 2015 at 4:57 PM, mkhaitman <ma...@chango.com> wrote:
> Hi Richard,
>
> Thanks for the response.
>
> I should have added that the specific case where this becomes a problem is
> when one of the executors for that application is lost/killed prematurely,
> and the application attempts to spawn up a new executor without
> consideration as to whether an executor already exists on the other node.
>
> In your example, if one of the executors dies for some reason (memory
> exhaustion, or something crashed it), if there are still free cores on the
> other nodes, it will spawn an extra executor, which can lead to further
> memory problems on the other node that it just spawned on.
>
> Hopefully that clears up what I mean :)
>
> Mark.
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Executor-Cores-question-tp14763p14805.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
--
*Richard Marscher*
Software Engineer
Localytics
Localytics.com <http://localytics.com/> | Our Blog
<http://localytics.com/blog> | Twitter <http://twitter.com/localytics> |
Facebook <http://facebook.com/localytics> | LinkedIn
<http://www.linkedin.com/company/1148792?trk=tyah>
Re: Spark.Executor.Cores question
Posted by mkhaitman <ma...@chango.com>.
Hi Richard,
Thanks for the response.
I should have added that the specific case where this becomes a problem is
when one of the executors for that application is lost/killed prematurely,
and the application attempts to spawn up a new executor without
consideration as to whether an executor already exists on the other node.
In your example, if one of the executors dies for some reason (memory
exhaustion, or something crashed it), if there are still free cores on the
other nodes, it will spawn an extra executor, which can lead to further
memory problems on the other node that it just spawned on.
Hopefully that clears up what I mean :)
Mark.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Executor-Cores-question-tp14763p14805.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark.Executor.Cores question
Posted by Richard Marscher <rm...@localytics.com>.
Hi Mark,
if you know your cluster's number of workers and cores per worker you can
set this up when you create a SparkContext and shouldn't need to tinker
with the 'spark.executor.cores' setting. That setting is for running
multiple executors per application per worker, which you are saying you
don't want.
How to do what I'm describing? In standalone mode, you will be assigned
cores in round robin order through the cluster's available workers (someone
correct me if that has changed since 1.3). So if you have 4 workers and set
`spark.cores.max` to `16` on your SparkContext then you will have 4
executors on each worker that are using 4 cores each. If you set
`spark.cores.max` to `6` then two executors would have 2 cores and two
executors would have 1 core.
Hope that helps
On Fri, Oct 23, 2015 at 3:05 PM, mkhaitman <ma...@chango.com> wrote:
> Regarding the 'spark.executor.cores' config option in a Standalone spark
> environment, I'm curious about whether there's a way to enforce the
> following logic:
>
> *- Max cores per executor = 4*
> ** Max executors PER application PER worker = 1*
>
> In order to force better balance across all workers, I want to ensure that
> a
> single spark job can only ever use a specific upper limit on the number of
> cores for each executor it holds, however, do not want a situation where it
> can spawn 3 executors on a worker and only 1/2 on the others. Some spark
> jobs end up using much more memory during aggregation tasks (joins /
> groupBy's) which is more heavily impacted by the number of cores per
> executor for that job.
>
> If this kind of setup/configuration doesn't already exist for Spark, and
> others see the benefit of what I mean by this, where would be the best
> location to insert this logic?
>
> Mark.
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Executor-Cores-question-tp14763.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
--
*Richard Marscher*
Software Engineer
Localytics
Localytics.com <http://localytics.com/> | Our Blog
<http://localytics.com/blog> | Twitter <http://twitter.com/localytics> |
Facebook <http://facebook.com/localytics> | LinkedIn
<http://www.linkedin.com/company/1148792?trk=tyah>