Tuning Spark memory

[tan shai <ta...@gmail.com>]

Hi,

I am working with Spark 2.0, the job starts by sorting the input data and
storing the output on HDFS.

I am getting Out of memory errors, the solution was to increase the value
of spark.shuffle.memoryFraction from 0.2 to 0.8 and this solves the
problem. But in the documentation I have found that this is a deprecated
parameter.

As I have understand, It was replaced by spark.memory.fraction. How to
modify this parameter while taking into account the sort and storage on
HDFS?

Thanks.

Re: Tuning Spark memory

[Takeshi Yamamuro <li...@gmail.com>]

Hi,

Currently, the memory fraction of shuffle and storage is automatically
tuned by a memory manager.
So, you do not need to care the parameter in most cases.
See
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L24

// maropu


On Fri, Sep 23, 2016 at 9:06 PM, tan shai <ta...@gmail.com> wrote:

> Hi,
>
> I am working with Spark 2.0, the job starts by sorting the input data and
> storing the output on HDFS.
>
> I am getting Out of memory errors, the solution was to increase the value
> of spark.shuffle.memoryFraction from 0.2 to 0.8 and this solves the
> problem. But in the documentation I have found that this is a deprecated
> parameter.
>
> As I have understand, It was replaced by spark.memory.fraction. How to
> modify this parameter while taking into account the sort and storage on
> HDFS?
>
> Thanks.
>



-- 
---
Takeshi Yamamuro