Regarding Off-heap memory

[Xiaoyu Ma <hz...@corp.netease.com>]

Hi all,
I saw spark 1.6 has new off heap settings: spark.memory.offHeap.size
The doc said we need to shrink on heap size accordingly. But on Yarn on-heap and yarn limit is set all together via spark.executor.memory (jvm opts for memory is not allowed according to doc), how can we set executor JVM heap size and Yarn container memory limit differently?

Thanks.

马晓宇
hzmaxiaoyu@corp.netease.com




---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Regarding Off-heap memory

[Nirav Patel <np...@xactlycorp.com>]

>From my experience with spark 1.3.1 you can also set
spark.executor.memoryOverhead to about 7-10% of your spark.executor.memory.
Total of which will be requested for a Yarn container.

On Tue, Jan 26, 2016 at 4:20 AM, Xiaoyu Ma <hz...@corp.netease.com>
wrote:

> Hi all,
> I saw spark 1.6 has new off heap settings: spark.memory.offHeap.size
> The doc said we need to shrink on heap size accordingly. But on Yarn
> on-heap and yarn limit is set all together via spark.executor.memory (jvm
> opts for memory is not allowed according to doc), how can we set executor
> JVM heap size and Yarn container memory limit differently?
>
> Thanks.
>
> 马晓宇
> hzmaxiaoyu@corp.netease.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

-- 


[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>

<https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn] 
<https://www.linkedin.com/company/xactly-corporation>  [image: Twitter] 
<https://twitter.com/Xactly>  [image: Facebook] 
<https://www.facebook.com/XactlyCorp>  [image: YouTube] 
<http://www.youtube.com/xactlycorporation>