You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2015/07/29 06:55:04 UTC

[jira] [Resolved] (SPARK-9419) ShuffleMemoryManager and MemoryStore should track memory on a per-task, not per-thread, basis

     [ https://issues.apache.org/jira/browse/SPARK-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Reynold Xin resolved SPARK-9419.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 1.5.0

> ShuffleMemoryManager and MemoryStore should track memory on a per-task, not per-thread, basis
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-9419
>                 URL: https://issues.apache.org/jira/browse/SPARK-9419
>             Project: Spark
>          Issue Type: Bug
>          Components: Block Manager, Spark Core
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>            Priority: Critical
>             Fix For: 1.5.0
>
>
> Spark's ShuffleMemoryManager and MemoryStore track memory on a per-thread basis, which causes problems in the handful of cases where we have tasks that use multiple threads. In PythonRDD, RRDD, ScriptTransformation, and PipedRDD we consume the input iterator in a separate thread in order to write it to an external process.  As a result, these RDD's input iterators are consumed in a different thread than the thread that created them, which can cause problems in our memory allocation tracking. For example, if allocations are performed in one thread but deallocations are performed in a separate thread then memory may be leaked or we may get errors complaining that more memory was allocated than was freed.
> I think that the right way to fix this is to change our accounting to be performed on a per-task instead of per-thread basis.  Note that the current per-thread tracking has caused problems in the past; SPARK-3731 (#2668) fixes a memory leak in PythonRDD that was caused by this issue (that fix is no longer necessary as of this patch).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org