[jira] [Created] (SPARK-13980) Incrementally serialize blocks while unrolling them in MemoryStore

["Josh Rosen (JIRA)" <ji...@apache.org>]

Josh Rosen created SPARK-13980:
----------------------------------

             Summary: Incrementally serialize blocks while unrolling them in MemoryStore
                 Key: SPARK-13980
                 URL: https://issues.apache.org/jira/browse/SPARK-13980
             Project: Spark
          Issue Type: Improvement
          Components: Block Manager
            Reporter: Josh Rosen
            Assignee: Josh Rosen


When a block is persisted in the MemoryStore at a serialized storage level, the current MemoryStore.putIterator() code will unroll the entire iterator as Java objects in memory, then will turn around and serialize an iterator obtained from the unrolled array. This is inefficient and doubles our peak memory requirements. Instead, I think that we should incrementally serialize blocks while unrolling them. A downside to incremental serialization is the fact that we will need to deserialize the partially-unrolled data in case there is not enough space to unroll the block and the block cannot be dropped to disk. However, I'm hoping that the memory efficiency improvements will outweigh any performance losses as a result of extra serialization in that hopefully-rare case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org