You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/10/06 21:33:27 UTC
[jira] [Assigned] (SPARK-10956) Introduce common memory management
interface for execution and storage
[ https://issues.apache.org/jira/browse/SPARK-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-10956:
------------------------------------
Assignee: Andrew Or (was: Apache Spark)
> Introduce common memory management interface for execution and storage
> ----------------------------------------------------------------------
>
> Key: SPARK-10956
> URL: https://issues.apache.org/jira/browse/SPARK-10956
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Critical
>
> Memory management in Spark is currently broken down into two disjoint regions: one for execution and one for storage. The sizes of these regions are statically configured and fixed for the duration of the application.
> There are several limitations to this approach. It requires user expertise to avoid unnecessary spilling, and there are no sensible defaults that will work for all workloads. Most importantly, applications that do not use caching use only a small fraction of the heap space, resulting in suboptimal performance.
> The first step towards implementing a solution is to refactor the existing code to go through a common MemoryManager interface. *This issue is concerned only with the introduction of this interface, preserving the existing behavior as much as possible.* In the near future, we will implement an alternate MemoryManager that shares memory between storage and execution more efficiently.
> A higher level design doc will be posted shortly on an upcoming issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org