You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Davies Liu (JIRA)" <ji...@apache.org> on 2015/08/28 23:03:45 UTC

[jira] [Created] (SPARK-10342) Cooperative memory management

Davies Liu created SPARK-10342:
----------------------------------

             Summary: Cooperative memory management
                 Key: SPARK-10342
                 URL: https://issues.apache.org/jira/browse/SPARK-10342
             Project: Spark
          Issue Type: Bug
          Components: Spark Core, SQL
    Affects Versions: 1.5.0
            Reporter: Davies Liu
            Assignee: Davies Liu
            Priority: Critical


We have memory starving problems for a long time, it become worser in 1.5 since we use larger page.

In order to increase the memory usage (reduce unnecessary spilling) also reduce the risk of OOM, we should manage the memory in a cooperative way, it means all the memory consume should be also responsive to release memory (spilling) upon others' requests.

The requests of memory could be different, hard requirement (will crash if not allocated) or soft requirement (worse performance if not allocated). Also the costs of spilling are also different. We could introduce some kind of priority to make them work together better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org