spark on mesos memory sizing with offheap

[vincent gromakowski <vi...@gmail.com>]

Hi,
I am trying to understand how mesos allocate memory when offheap is enabled
but it seems that the framework is only taking the heap + 400 MB overhead
into consideration for resources allocation.
Example: spark.executor.memory=3g spark.memory.offheap.size=1g ==> mesos
report 3.4g allocated for the executor
Is there any configuration to use both heap and offheap for mesos
allocation ?

Re: spark on mesos memory sizing with offheap

[Michael Gummelt <mg...@mesosphere.io>]

It doesn't look like we are.  Can you file a JIRA?  A workaround is to set
spark.mesos.executor.overhead to be at least spark.memory.offheap.size.
This is how the container is sized:
https://github.com/apache/spark/blob/master/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala#L366

On Thu, Oct 13, 2016 at 7:23 AM, vincent gromakowski <
vincent.gromakowski@gmail.com> wrote:

> Hi,
> I am trying to understand how mesos allocate memory when offheap is
> enabled but it seems that the framework is only taking the heap + 400 MB
> overhead into consideration for resources allocation.
> Example: spark.executor.memory=3g spark.memory.offheap.size=1g ==> mesos
> report 3.4g allocated for the executor
> Is there any configuration to use both heap and offheap for mesos
> allocation ?
>



-- 
Michael Gummelt
Software Engineer
Mesosphere