[jira] [Created] (SPARK-10956) Introduce common memory management interface for execution and storage

["Andrew Or (JIRA)" <ji...@apache.org>]

Andrew Or created SPARK-10956:
---------------------------------

             Summary: Introduce common memory management interface for execution and storage
                 Key: SPARK-10956
                 URL: https://issues.apache.org/jira/browse/SPARK-10956
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 1.0.0
            Reporter: Andrew Or
            Assignee: Andrew Or
            Priority: Critical


Memory management in Spark is currently broken down into two disjoint regions: one for execution and one for storage. The sizes of these regions are statically configured and fixed for the duration of the application.

There are several limitations to this approach. It requires user expertise to avoid unnecessary spilling, and there are no sensible defaults that will work for all workloads. Most importantly, applications that do not use caching use only a small fraction of the heap space, resulting in suboptimal performance.

The first step towards implementing a solution is to refactor the existing code to go through a common MemoryManager interface. *This issue is concerned only with the introduction of this interface, preserving the existing behavior as much as possible.* In the near future, we will implement an alternate MemoryManager that shares memory between storage and execution more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org