You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "zhoukang (JIRA)" <ji...@apache.org> on 2018/01/17 09:25:00 UTC

[jira] [Created] (SPARK-23129) Lazy init DiskMapIterator#deserializeStream to reduce memory usage when ExternalAppendOnlyMap spill too much times

zhoukang created SPARK-23129:
--------------------------------

             Summary: Lazy init DiskMapIterator#deserializeStream to reduce memory usage when ExternalAppendOnlyMap spill  too much times
                 Key: SPARK-23129
                 URL: https://issues.apache.org/jira/browse/SPARK-23129
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.1.0
            Reporter: zhoukang


Currently,the deserializeStream in ExternalAppendOnlyMap#DiskMapIterator init when DiskMapIterator instance created.This will cause memory use overhead when ExternalAppendOnlyMap spill too much times.

We can avoid this by making deserializeStream init when it is used the first time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org